Updated changes to 3.1.1.

Fixed issue #2808 - Unprocessable mail if follow up search is enabled for attachments with contains invalid charsets (postmaster_follow_up_search_in with attachment).
Fixed issue #2796 - Image/attachment preview is to small, need to download image and open it locally.
2019-11-29 15:17:25 +01:00 · 2019-11-18 18:53:39 +01:00 · 2019-11-18 18:50:40 +01:00 · 2019-11-18 17:04:32 +01:00 · 2019-11-18 16:32:41 +01:00 · 2019-11-07 13:30:41 +01:00
53 changed files with 1356 additions and 585 deletions
--- a/.circleci/install.sh
+++ b/.circleci/install.sh
@@ -33,7 +33,7 @@ bundle install --without "${INSTALL_OPTION}"
 bundle exec rubocop
 rake db:migrate
 rake db:seed
-bundle exec rspec -t ~type:system
+bundle exec rspec -t ~type:system -t ~searchindex
 bundle exec rake db:environment:set RAILS_ENV=test
 rake db:reset
 rake test:units
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -121,7 +121,7 @@ pre:github:
    RAILS_ENV: "test"
  script:
    - bundle exec rake zammad:db:init
-    - bundle exec rspec -t ~type:system
+    - bundle exec rspec -t ~type:system -t ~searchindex

 test:rspec:mysql:
  stage: test
--- a/.pkgr.yml
+++ b/.pkgr.yml
@@ -38,6 +38,17 @@ targets:
    build_dependencies:
      - libimlib2
      - libimlib2-dev
+  debian-10:
+    dependencies:
+      - curl
+      - elasticsearch
+      - nginx|apache2
+      - postgresql|mariadb-server|sqlite
+      - libimlib2
+      - libimlib2-dev
+    build_dependencies:
+      - libimlib2
+      - libimlib2-dev
  ubuntu-16.04:
    dependencies:
      - curl
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,15 +1,52 @@
 # Change Log

-## [3.1.0](https://github.com/zammad/zammad/tree/3.1.0) (2019-xx-xx)
+## [3.1.1](https://github.com/zammad/zammad/tree/3.1.1) (2019-12-02)
+[Full Changelog](https://github.com/zammad/zammad/compare/3.1.0...3.1.1)
+
+**Implemented enhancements:**
+- Sometimes elasticsearch takes longer as expected and will raise timeout in Zammad [2720](https://github.com/zammad/zammad/issues/2720) [[enhancement](https://github.com/zammad/zammad/labels/enhancement)]
+
+**Fixed bugs:**
+- Unprocessable mail if follow up search is enabled for attachments with contains invalid charsets (postmaster_follow_up_search_in with attachment) [2808](https://github.com/zammad/zammad/issues/2808) [[bug](https://github.com/zammad/zammad/labels/bug)] [[mail processing](https://github.com/zammad/zammad/labels/mail processing)]
+- Image/attachment preview is to small, need to download image and open it locally  [2796](https://github.com/zammad/zammad/issues/2796) [[UX/UI](https://github.com/zammad/zammad/labels/UX/UI)]
+- Unable to process email where group has option `follow_up_possible: 'new_ticket'` and arraiving email has no subject header [2768](https://github.com/zammad/zammad/issues/2768) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Fixes #2699 - Excel export of report or time accounting drops additional fields in row if integer field contain `null/nil/undefined`. [2699](https://github.com/zammad/zammad/issues/2699) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Removing a record drops whole index on Elasticsearch 6 and later [2742](https://github.com/zammad/zammad/issues/2742) [[bug](https://github.com/zammad/zammad/labels/bug)] [[knowledge base](https://github.com/zammad/zammad/labels/knowledge base)]
+- Can't change Ticket attributes of "Shared Organization" Ticket [2750](https://github.com/zammad/zammad/issues/2750) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Put _type, _doc and _id to reserved words for object managaer becasue of elasticssearch internal use [2728](https://github.com/zammad/zammad/issues/2728) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Unable to process mail with enabled follow up detection in content of attachments [2723](https://github.com/zammad/zammad/issues/2723) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- mail processing impossible: invalid byte sequence in UTF-8 [2722](https://github.com/zammad/zammad/issues/2722) [[bug](https://github.com/zammad/zammad/labels/bug)] [[mail processing](https://github.com/zammad/zammad/labels/mail processing)]
+- Unable to set owner via `X-Zammad-Ticket-owner` header [2719](https://github.com/zammad/zammad/issues/2719) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Time zone incorrectly calculated in ticket list [2697](https://github.com/zammad/zammad/issues/2697) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Linking of external account in profile is broken [2675](https://github.com/zammad/zammad/issues/2675) [[authentication](https://github.com/zammad/zammad/labels/authentication)] [[bug](https://github.com/zammad/zammad/labels/bug)] [[personal settings/menu](https://github.com/zammad/zammad/labels/personal settings/menu)]
+- Issue2541FixNotificationEmailWithoutBody database migration blocks update when records with `current_user.id` in selector exist [2663](https://github.com/zammad/zammad/issues/2663) [[bug](https://github.com/zammad/zammad/labels/bug)] [[update](https://github.com/zammad/zammad/labels/update)]
+- Zammad logs permanently to STDOUT on 3.1 [2660](https://github.com/zammad/zammad/issues/2660) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Last line in note is not selectable [2654](https://github.com/zammad/zammad/issues/2654) [[UX/UI](https://github.com/zammad/zammad/labels/UX/UI)] [[bug](https://github.com/zammad/zammad/labels/bug)] [[ticket](https://github.com/zammad/zammad/labels/ticket)]
+- Services die with: ThreadError: can't create Thread: Resource temporarily unavailable [2656](https://github.com/zammad/zammad/issues/2656) [[bug](https://github.com/zammad/zammad/labels/bug)] [[third party issue](https://github.com/zammad/zammad/labels/third party issue)] [[waiting for feedback](https://github.com/zammad/zammad/labels/waiting for feedback)]
+
+
+## [3.1.0](https://github.com/zammad/zammad/tree/3.1.0) (2019-07-10)
 [Full Changelog](https://github.com/zammad/zammad/compare/3.0.0...3.1.0)

 **Implemented enhancements:**
-
-
-
+- Enhance tag search to use fulltext search [2569](https://github.com/zammad/zammad/issues/2569) [[UX/UI](https://github.com/zammad/zammad/labels/UX/UI)] [[enhancement](https://github.com/zammad/zammad/labels/enhancement)] [[ticket](https://github.com/zammad/zammad/labels/ticket)]
+- translation strings wrong [1561](https://github.com/zammad/zammad/issues/1561) [[enhancement](https://github.com/zammad/zammad/labels/enhancement)] [[translation](https://github.com/zammad/zammad/labels/translation)]
+- Misleading 'Escalation' ticket attribute name in SLAs, Scheduler and Trigger should be named 'Escalation_at' [2615](https://github.com/zammad/zammad/issues/2615) [[enhancement](https://github.com/zammad/zammad/labels/enhancement)] [[trigger](https://github.com/zammad/zammad/labels/trigger)]
+- Add scroll bar to tag list if not enough room is available [2570](https://github.com/zammad/zammad/issues/2570) [[UX/UI](https://github.com/zammad/zammad/labels/UX/UI)] [[enhancement](https://github.com/zammad/zammad/labels/enhancement)]
+- Elasticsearch 6.x & 7.x support [1688](https://github.com/zammad/zammad/issues/1688) [[enhancement](https://github.com/zammad/zammad/labels/enhancement)]

 **Fixed bugs:**
-
-
-
-
+- SearchIndexBackend.remove() (ES) is not working correctly [2611](https://github.com/zammad/zammad/issues/2611) [[bug](https://github.com/zammad/zammad/labels/bug)] [[search](https://github.com/zammad/zammad/labels/search)]
+- Add Rails 5 log to STDOUT env variable support [2637](https://github.com/zammad/zammad/pull/2637)
+- Removed duplicate `Mailing-List` tag from postmaster match ui element [2639](https://github.com/zammad/zammad/pull/2639) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Integer minimal is ignored [2339](https://github.com/zammad/zammad/issues/2339) [[bug](https://github.com/zammad/zammad/labels/bug)] [[object manager attribute](https://github.com/zammad/zammad/labels/object manager attribute)]
+- Setup asks for FQDN, but doesn't write to DB [462](https://github.com/zammad/zammad/issues/462) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Concern 'CanLatestChange' returns wrong updated_at  [2624](https://github.com/zammad/zammad/issues/2624) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- KB partly broken on IE11 - customer interface [2600](https://github.com/zammad/zammad/issues/2600) [[bug](https://github.com/zammad/zammad/labels/bug)] [[knowledge base](https://github.com/zammad/zammad/labels/knowledge base)]
+- missing OTRS.png asset in Import Wizard [2593](https://github.com/zammad/zammad/issues/2593) [[bug](https://github.com/zammad/zammad/labels/bug)] [[frontend / JS app](https://github.com/zammad/zammad/labels/frontend / JS app)] [[import](https://github.com/zammad/zammad/labels/import)]
+- Wrong date format in Excel Exports of reporting download [2594](https://github.com/zammad/zammad/issues/2594) [[bug](https://github.com/zammad/zammad/labels/bug)] [[reporting](https://github.com/zammad/zammad/labels/reporting)]
+- Tree Select shows empty value in ticket zoom after submit when value contains trailing spaces [2614](https://github.com/zammad/zammad/issues/2614) [[bug](https://github.com/zammad/zammad/labels/bug)] [[object manager attribute](https://github.com/zammad/zammad/labels/object manager attribute)]
+- Edit-Buttons on customer interface (as agent) has wrong link (Blank-Page) [2604](https://github.com/zammad/zammad/issues/2604) [[bug](https://github.com/zammad/zammad/labels/bug)] [[knowledge base](https://github.com/zammad/zammad/labels/knowledge base)]
+- Activation of Knowledgebase is impossible on IE11 - agent interface [2599](https://github.com/zammad/zammad/issues/2599) [[bug](https://github.com/zammad/zammad/labels/bug)] [[knowledge base](https://github.com/zammad/zammad/labels/knowledge base)]
+- Existing scheduler email notification without `body` will raise an exception. [2541](https://github.com/zammad/zammad/issues/2541) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- trigger email notification UI defective on removal of recipient [2516](https://github.com/zammad/zammad/issues/2516) [[UX/UI](https://github.com/zammad/zammad/labels/UX/UI)] [[bug](https://github.com/zammad/zammad/labels/bug)] [[frontend / JS app](https://github.com/zammad/zammad/labels/frontend / JS app)]
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -158,7 +158,7 @@ GEM
      tins (~> 1.6)
    crack (0.4.3)
      safe_yaml (~> 1.0.0)
-    crass (1.0.4)
+    crass (1.0.5)
    daemons (1.3.1)
    dalli (2.7.10)
    debug_inspector (0.0.3)
@@ -270,7 +270,7 @@ GEM
    logging (2.2.2)
      little-plugger (~> 1.1)
      multi_json (~> 1.10)
-    loofah (2.2.3)
+    loofah (2.3.1)
      crass (~> 1.0.2)
      nokogiri (>= 1.5.9)
    lumberjack (1.0.13)
@@ -299,7 +299,7 @@ GEM
    net-ldap (0.16.1)
    netrc (0.11.0)
    nio4r (2.3.1)
-    nokogiri (1.10.3)
+    nokogiri (1.10.5)
      mini_portile2 (~> 2.4.0)
    nori (2.6.0)
    notiffany (0.1.1)
@@ -454,7 +454,7 @@ GEM
      rubocop (>= 0.60.0)
    ruby-progressbar (1.10.0)
    ruby_dep (1.5.0)
-    rubyzip (1.2.2)
+    rubyzip (1.3.0)
    safe_yaml (1.0.5)
    sassc (2.0.1)
      ffi (~> 1.9)
--- a/app/assets/javascripts/app/controllers/_application_controller.coffee
+++ b/app/assets/javascripts/app/controllers/_application_controller.coffee
@@ -444,6 +444,7 @@ class App.ControllerModal extends App.Controller
  keyboard: true
  large: false
  small: false
+  veryLarge: false
  head: '?'
  autoFocusOnFirstInput: true
  container: null
--- a/app/assets/javascripts/app/controllers/_profile/linked_accounts.coffee
+++ b/app/assets/javascripts/app/controllers/_profile/linked_accounts.coffee
@@ -2,6 +2,7 @@ class Index extends App.ControllerSubContent
  requiredPermission: 'user_preferences.linked_accounts'
  header: 'Linked Accounts'
  events:
+    'click .js-add':    'add'
    'click .js-remove': 'remove'

  constructor: ->
@@ -20,6 +21,11 @@ class Index extends App.ControllerSubContent
      auth_providers: auth_providers
    )

+  add: (e) =>
+    e.preventDefault()
+    key = $(e.target).data('key')
+    @el.find(".js-addForm-#{key}").submit()
+
  remove: (e) =>
    e.preventDefault()
    provider = $(e.target).data('provider')
--- a/app/assets/javascripts/app/controllers/ticket_zoom/article_image_view.coffee
+++ b/app/assets/javascripts/app/controllers/ticket_zoom/article_image_view.coffee
@@ -4,7 +4,7 @@ class App.TicketZoomArticleImageView extends App.ControllerModal
  buttonSubmit: 'Download'
  buttonClass: 'btn--success'
  head: ''
-  large: true
+  veryLarge: true

  events:
    'submit form':      'submit'
--- a/app/assets/javascripts/app/models/ticket.coffee
+++ b/app/assets/javascripts/app/models/ticket.coffee
@@ -266,6 +266,8 @@ class App.Ticket extends App.Model
    user = App.User.current()
    return false if !user?
    return true if user.id is @customer_id
+    return true if user.organization_id && @organization_id && user.organization_id is @organization_id
+    return false if !@group_id
    group_ids = user.allGroupIds(permission)
    for local_group_id in group_ids
      if local_group_id.toString() is @group_id.toString()
--- a/app/assets/javascripts/app/views/profile/linked_accounts.jst.eco
+++ b/app/assets/javascripts/app/views/profile/linked_accounts.jst.eco
@@ -7,12 +7,16 @@
  <ul>
    <% for key, provider of @auth_providers: %>
      <li> <%- @T( provider.name ) %>
-        <% if !@user['accounts'] || !@user['accounts'][key]: %><a href="<%= provider.url %>">
-          <%- @T('Add') %></a>
+        <% if !@user['accounts'] || !@user['accounts'][key]: %>
+          <form method="post" class="hidden js-addForm-<%= key %>" action="<%= provider.url %>">
+            <input type="hidden" name="authenticity_token" value="<%= Spine.Ajax.defaults.headers['X-CSRF-Token'] %>">
+          </form>
+
+          <a href="#" data-key="<%= key %>" class="js-add"><%- @T('Add') %></a>
        <% else: %>
          "<%= @user['accounts'][key]['username'] %>" <a href="#" data-uid="<%= @user['accounts'][key]['uid'] %>" data-provider="<%= key %>" class="js-remove"><%- @T('remove') %></a>
        <% end %>
      </li>
    <% end %>
  </ul>
-</div>
+</div>
--- a/app/assets/stylesheets/zammad.scss
+++ b/app/assets/stylesheets/zammad.scss
@@ -5608,13 +5608,13 @@ footer {

    &.is-open {
      &:before {
-        opacity: 0;
+        display: none;
      }
    }

    .btn {
-      padding-top: 17px;
-      padding-bottom: 17px;
+      padding-top: 15px;
+      padding-bottom: 15px;
      font-size: 10px;
    }

@@ -5626,7 +5626,7 @@ footer {
      top: -30px;
      height: 30px;
      background: linear-gradient(rgba(255,255,255,0), white);
-      transition: opacity 300ms;
+      pointer-events: none;
    }
  }

--- a/app/controllers/reports_controller.rb
+++ b/app/controllers/reports_controller.rb
@@ -61,7 +61,7 @@ class ReportsController < ApplicationController

    # get data
    result = {}
-    content = nil
+    excel = nil
    filename = nil
    get_params[:metric][:backend].each do |backend|
      next if params[:downloadBackendSelected] != backend[:name]
@@ -89,14 +89,19 @@ class ReportsController < ApplicationController

      # generate sheet
      if params[:sheet]
-        content = sheet(get_params[:profile], backend[:display], result)
+        excel = ExcelSheet::Ticket.new(
+          title:      "#{get_params[:profile].name} (#{backend[:display]})",
+          ticket_ids: result[:ticket_ids],
+          timezone:   params[:timezone],
+          locale:     current_user.locale,
+        )
        filename = "tickets-#{get_params[:profile].name}-#{backend[:display]}.xls"
      end
      break
    end
-    if content
+    if excel
      send_data(
-        content,
+        excel.content,
        filename:    filename,
        type:        'application/vnd.ms-excel',
        disposition: 'attachment'
@@ -176,147 +181,4 @@ class ReportsController < ApplicationController
    }
  end

-  def sheet(profile, title, result)
-
-    params[:timezone] ||= Setting.get('timezone_default')
-
-    # Create a new Excel workbook
-    temp_file = Tempfile.new('time_tracking.xls')
-    workbook = WriteExcel.new(temp_file)
-
-    # Add a worksheet
-    worksheet = workbook.add_worksheet
-    worksheet.set_row(0, 18)
-    worksheet.set_column(0, 0, 10)
-    worksheet.set_column(1, 1, 34)
-    worksheet.set_column(2, 2, 10)
-    worksheet.set_column(3, 3, 10)
-    worksheet.set_column(4, 8, 20)
-    worksheet.set_column(11, 0, 20)
-    worksheet.set_column(12, 0, 20)
-    worksheet.set_column(13, 0, 20)
-
-    # Add and define a format
-    format = workbook.add_format
-    format.set_bold
-    format.set_size(14)
-    format.set_color('black')
-
-    # Write a formatted and unformatted string, row and column notation.
-    worksheet.write_string(0, 0, "Tickets: #{profile.name} (#{title})", format)
-
-    format_header = workbook.add_format
-    format_header.set_italic
-    format_header.set_bg_color('gray')
-    format_header.set_color('white')
-
-    format_time = workbook.add_format(num_format: 'yyyy-mm-dd hh:mm:ss')
-    format_date = workbook.add_format(num_format: 'yyyy-mm-dd')
-
-    format_footer = workbook.add_format
-    format_footer.set_italic
-    format_footer.set_color('gray')
-    format_footer.set_size(8)
-
-    worksheet.write_string(2, 0, '#', format_header)
-    worksheet.write_string(2, 1, 'Title', format_header)
-    worksheet.write_string(2, 2, 'State', format_header)
-    worksheet.write_string(2, 3, 'Priority', format_header)
-    worksheet.write_string(2, 4, 'Group', format_header)
-    worksheet.write_string(2, 5, 'Owner', format_header)
-    worksheet.write_string(2, 6, 'Customer', format_header)
-    worksheet.write_string(2, 7, 'Organization', format_header)
-    worksheet.write_string(2, 8, 'Create Channel', format_header)
-    worksheet.write_string(2, 9, 'Sender', format_header)
-    worksheet.write_string(2, 10, 'Tags', format_header)
-    worksheet.write_string(2, 11, 'Created at', format_header)
-    worksheet.write_string(2, 12, 'Updated at', format_header)
-    worksheet.write_string(2, 13, 'Closed at', format_header)
-
-    # ObjectManager attributes
-    header_column = 14
-    # needs to be skipped
-    objects = ObjectManager::Attribute.where(editable:         true,
-                                             active:           true,
-                                             to_create:        false,
-                                             object_lookup_id: ObjectLookup.lookup(name: 'Ticket').id)
-                                      .pluck(:name, :display, :data_type, :data_option)
-                                      .map { |name, display, data_type, data_option| { name: name, display: display, data_type: data_type, data_option: data_option } }
-    objects.each do |object|
-      worksheet.set_column(header_column, 0, 16)
-      worksheet.write_string(2, header_column, object[:display].capitalize, format_header)
-      header_column += 1
-    end
-
-    row = 2
-    result[:ticket_ids].each do |ticket_id|
-
-      ticket = Ticket.lookup(id: ticket_id)
-      row += 1
-      worksheet.write_string(row, 0, ticket.number)
-      worksheet.write_string(row, 1, ticket.title)
-      worksheet.write_string(row, 2, ticket.state.name)
-      worksheet.write_string(row, 3, ticket.priority.name)
-      worksheet.write_string(row, 4, ticket.group.name)
-      worksheet.write_string(row, 5, ticket.owner.fullname)
-      worksheet.write_string(row, 6, ticket.customer.fullname)
-      worksheet.write_string(row, 7, ticket.try(:organization).try(:name))
-      worksheet.write_string(row, 8, ticket.create_article_type.name)
-      worksheet.write_string(row, 9, ticket.create_article_sender.name)
-      worksheet.write_string(row, 10, ticket.tag_list.join(','))
-
-      worksheet.write_date_time(row, 11, time_in_localtime_for_excel(ticket.created_at, params[:timezone]), format_time)
-      worksheet.write_date_time(row, 12, time_in_localtime_for_excel(ticket.updated_at, params[:timezone]), format_time)
-      worksheet.write_date_time(row, 13, time_in_localtime_for_excel(ticket.close_at, params[:timezone]), format_time) if ticket.close_at.present?
-
-      # Object Manager attributes
-      column = 14
-      # We already queried ObjectManager::Attributes, so we just use objects
-      objects.each do |object|
-        key = object[:name]
-        case object[:data_type]
-        when 'boolean', 'select'
-          value = ticket.send(key.to_sym)
-          if object[:data_option] && object[:data_option]['options'] && object[:data_option]['options'][ticket.send(key.to_sym)]
-            value = object[:data_option]['options'][ticket.send(key.to_sym)]
-          end
-          worksheet.write_string(row, column, value)
-        when 'datetime'
-          worksheet.write_date_time(row, column, time_in_localtime_for_excel(ticket.send(key.to_sym), params[:timezone]), format_time) if ticket.send(key.to_sym).present?
-        when 'date'
-          worksheet.write_date_time(row, column, ticket.send(key.to_sym).to_s, format_date) if ticket.send(key.to_sym).present?
-        when 'integer'
-          worksheet.write_number(row, column, ticket.send(key.to_sym))
-        else
-          # for text, integer and tree select
-          worksheet.write_string(row, column, ticket.send(key.to_sym).to_s)
-        end
-        column += 1
-      end
-    rescue => e
-      Rails.logger.error "SKIP: #{e.message}"
-
-    end
-
-    row += 2
-    worksheet.write_string(row, 0, "#{Translation.translate(current_user.locale, 'Timezone')}: #{params[:timezone]}", format_footer)
-
-    workbook.close
-
-    # read file again
-    file = File.new(temp_file, 'r')
-    contents = file.read
-    file.close
-    contents
-  end
-
-  def time_in_localtime_for_excel(time, timezone)
-    return if time.blank?
-
-    if timezone.present?
-      offset = time.in_time_zone(timezone).utc_offset
-      time -= offset
-    end
-    time.utc.iso8601.to_s.sub(/Z$/, '')
-  end
 end
--- a/app/controllers/time_accountings_controller.rb
+++ b/app/controllers/time_accountings_controller.rb
@@ -21,216 +21,79 @@ class TimeAccountingsController < ApplicationController
      end
      time_unit[record[0]][:time_unit] += record[1]
    end
-    customers = {}
-    organizations = {}
-    agents = {}
-    results = []
-    time_unit.each do |ticket_id, local_time_unit|
-      ticket = Ticket.lookup(id: ticket_id)
-      next if !ticket

-      if !customers[ticket.customer_id]
-        customers[ticket.customer_id] = '-'
-        if ticket.customer_id
-          customer_user = User.lookup(id: ticket.customer_id)
-          if customer_user
-            customers[ticket.customer_id] = customer_user.fullname
-          end
-        end
-      end
-      if !organizations[ticket.organization_id]
-        organizations[ticket.organization_id] = '-'
-        if ticket.organization_id
-          organization = Organization.lookup(id: ticket.organization_id)
-          if organization
-            organizations[ticket.organization_id] = organization.name
-          end
-        end
-      end
-      if !agents[local_time_unit[:agent_id]]
-        agent_user = User.lookup(id: local_time_unit[:agent_id])
-        if agent_user
-          agents[local_time_unit[:agent_id]] = agent_user.fullname
-        end
-      end
-      result = {
-        ticket:       ticket.attributes,
-        time_unit:    local_time_unit[:time_unit],
-        customer:     customers[ticket.customer_id],
-        organization: organizations[ticket.organization_id],
-        agent:        agents[local_time_unit[:agent_id]],
-      }
-      results.push result
-    end
+    if !params[:download]
+      customers = {}
+      organizations = {}
+      agents = {}
+      results = []
+      time_unit.each do |ticket_id, local_time_unit|
+        ticket = Ticket.lookup(id: ticket_id)
+        next if !ticket

-    if params[:download]
-      header = [
-        {
-          name:  'Ticket#',
-          width: 15,
-        },
-        {
-          name:  'Title',
-          width: 30,
-        },
-        {
-          name:  'Customer',
-          width: 20,
-        },
-        {
-          name:  'Organization',
-          width: 20,
-        },
-        {
-          name:  'Agent',
-          width: 20,
-        },
-        {
-          name:  'Time Units',
-          width: 10,
-        },
-        {
-          name:  'Time Units Total',
-          width: 10,
-        },
-        {
-          name:  'Created at',
-          width: 18,
-        },
-        {
-          name:  'Closed at',
-          width: 18,
-        },
-        {
-          name:  'Close Escalation At',
-          width: 18,
-        },
-        {
-          name:  'Close In Min',
-          width: 10,
-        },
-        {
-          name:  'Close Diff In Min',
-          width: 10,
-        },
-        {
-          name:  'First Response At',
-          width: 18,
-        },
-        {
-          name:  'First Response Escalation At',
-          width: 18,
-        },
-        {
-          name:  'First Response In Min',
-          width: 10,
-        },
-        {
-          name:  'First Response Diff In Min',
-          width: 10,
-        },
-        {
-          name:  'Update Escalation At',
-          width: 18,
-        },
-        {
-          name:  'Update In Min',
-          width: 10,
-        },
-        {
-          name:  'Update Diff In Min',
-          width: 10,
-        },
-        {
-          name:  'Last Contact At',
-          width: 18,
-        },
-        {
-          name:  'Last Contact Agent At',
-          width: 18,
-        },
-        {
-          name:  'Last Contact Customer At',
-          width: 18,
-        },
-        {
-          name:  'Article Count',
-          width: 10,
-        },
-        {
-          name:  'Escalation At',
-          width: 18,
-        },
-      ]
-      objects = ObjectManager::Attribute.where(editable:         true,
-                                               active:           true,
-                                               to_create:        false,
-                                               object_lookup_id: ObjectLookup.lookup(name: 'Ticket').id)
-                                        .pluck(:name, :display, :data_type, :data_option)
-                                        .map { |name, display, data_type, data_option| { name: name, display: display, data_type: data_type, data_option: data_option } }
-      objects.each do |object|
-        header.push({ name: object[:display], width: 18 })
-      end
-
-      result = []
-      results.each do |row|
-        result_row = [
-          row[:ticket]['number'],
-          row[:ticket]['title'],
-          row[:customer],
-          row[:organization],
-          row[:agent],
-          row[:time_unit],
-          row[:ticket]['time_unit'],
-          row[:ticket]['created_at'],
-          row[:ticket]['close_at'],
-          row[:ticket]['close_escalation_at'],
-          row[:ticket]['close_in_min'],
-          row[:ticket]['close_diff_in_min'],
-          row[:ticket]['first_response_at'],
-          row[:ticket]['first_response_escalation_at'],
-          row[:ticket]['first_response_in_min'],
-          row[:ticket]['first_response_diff_in_min'],
-          row[:ticket]['update_escalation_at'],
-          row[:ticket]['update_in_min'],
-          row[:ticket]['update_diff_in_min'],
-          row[:ticket]['last_contact_at'],
-          row[:ticket]['last_contact_agent_at'],
-          row[:ticket]['last_contact_customer_at'],
-          row[:ticket]['article_count'],
-          row[:ticket]['escalation_at'],
-        ]
-
-        # Object Manager attributes
-        # We already queried ObjectManager::Attributes, so we just use objects
-        objects.each do |object|
-          key = object[:name]
-          case object[:data_type]
-          when 'boolean', 'select'
-            value = row[:ticket][key]
-            if object[:data_option] && object[:data_option]['options'] && object[:data_option]['options'][row[:ticket][key]]
-              value = object[:data_option]['options'][row[:ticket][key]]
+        if !customers[ticket.customer_id]
+          customers[ticket.customer_id] = '-'
+          if ticket.customer_id
+            customer_user = User.lookup(id: ticket.customer_id)
+            if customer_user
+              customers[ticket.customer_id] = customer_user.fullname
            end
-            value.present? ? result_row.push(value) : result_row.push('')
-          else
-            # for text, integer and tree select
-            row[:ticket][key].present? ? result_row.push(row[:ticket][key]) : result_row.push('')
          end
        end
-
-        result.push result_row
+        if !organizations[ticket.organization_id]
+          organizations[ticket.organization_id] = '-'
+          if ticket.organization_id
+            organization = Organization.lookup(id: ticket.organization_id)
+            if organization
+              organizations[ticket.organization_id] = organization.name
+            end
+          end
+        end
+        if !agents[local_time_unit[:agent_id]]
+          agent_user = User.lookup(id: local_time_unit[:agent_id])
+          if agent_user
+            agents[local_time_unit[:agent_id]] = agent_user.fullname
+          end
+        end
+        result = {
+          ticket:       ticket.attributes,
+          time_unit:    local_time_unit[:time_unit],
+          customer:     customers[ticket.customer_id],
+          organization: organizations[ticket.organization_id],
+          agent:        agents[local_time_unit[:agent_id]],
+        }
+        results.push result
      end
-      content = sheet("By Ticket #{year}-#{month}", header, result)
-      send_data(
-        content,
-        filename:    "by_ticket-#{year}-#{month}.xls",
-        type:        'application/vnd.ms-excel',
-        disposition: 'attachment'
-      )
+      render json: results
      return
    end

-    render json: results
+    ticket_ids = []
+    additional_attributes = []
+    additional_attributes_header = [{ display: 'Time Units', name: 'time_unit_for_range', width: 10, data_type: 'float' }]
+    time_unit.each do |ticket_id, local_time_unit|
+      ticket_ids.push ticket_id
+      additional_attribute = {
+        time_unit_for_range: local_time_unit[:time_unit],
+      }
+      additional_attributes.push additional_attribute
+    end
+
+    excel = ExcelSheet::Ticket.new(
+      title:                        "Tickets: #{year}-#{month}",
+      ticket_ids:                   ticket_ids,
+      additional_attributes:        additional_attributes,
+      additional_attributes_header: additional_attributes_header,
+      timezone:                     params[:timezone],
+      locale:                       current_user.locale,
+    )
+
+    send_data(
+      excel.content,
+      filename:    "by_ticket-#{year}-#{month}.xls",
+      type:        'application/vnd.ms-excel',
+      disposition: 'attachment'
+    )
  end

  def by_customer
@@ -287,11 +150,12 @@ class TimeAccountingsController < ApplicationController
          width: 30,
        },
        {
-          name:  'Time Units',
-          width: 10,
+          name:      'Time Units',
+          width:     10,
+          data_type: 'float'
        }
      ]
-      result = []
+      records = []
      results.each do |row|
        customer_name = User.find(row[:customer]['id']).fullname
        organization_name = ''
@@ -299,11 +163,18 @@ class TimeAccountingsController < ApplicationController
          organization_name = row[:organization]['name']
        end
        result_row = [customer_name, organization_name, row[:time_unit]]
-        result.push result_row
+        records.push result_row
      end
-      content = sheet("By Customer #{year}-#{month}", header, result)
+
+      excel = ExcelSheet.new(
+        title:    "By Customer #{year}-#{month}",
+        header:   header,
+        records:  records,
+        timezone: params[:timezone],
+        locale:   current_user.locale,
+      )
      send_data(
-        content,
+        excel.content,
        filename:    "by_customer-#{year}-#{month}.xls",
        type:        'application/vnd.ms-excel',
        disposition: 'attachment'
@@ -360,22 +231,30 @@ class TimeAccountingsController < ApplicationController
          width: 40,
        },
        {
-          name:  'Time Units',
-          width: 20,
+          name:      'Time Units',
+          width:     20,
+          data_type: 'float',
        }
      ]
-      result = []
+      records = []
      results.each do |row|
        organization_name = ''
        if row[:organization].present?
          organization_name = row[:organization]['name']
        end
        result_row = [organization_name, row[:time_unit]]
-        result.push result_row
+        records.push result_row
      end
-      content = sheet("By Organization #{year}-#{month}", header, result)
+
+      excel = ExcelSheet.new(
+        title:    "By Organization #{year}-#{month}",
+        header:   header,
+        records:  records,
+        timezone: params[:timezone],
+        locale:   current_user.locale,
+      )
      send_data(
-        content,
+        excel.content,
        filename:    "by_organization-#{year}-#{month}.xls",
        type:        'application/vnd.ms-excel',
        disposition: 'attachment'
@@ -385,89 +264,4 @@ class TimeAccountingsController < ApplicationController

    render json: results
  end
-
-  private
-
-  def sheet(title, header, result)
-
-    params[:timezone] ||= Setting.get('timezone_default')
-
-    # Create a new Excel workbook
-    temp_file = Tempfile.new('time_tracking.xls')
-    workbook = WriteExcel.new(temp_file)
-
-    # Add a worksheet
-    worksheet = workbook.add_worksheet
-
-    # Add and define a format
-    format = workbook.add_format  # Add a format
-    format.set_bold
-    format.set_size(14)
-    format.set_color('black')
-
-    format_time = workbook.add_format(num_format: 'yyyy-mm-dd hh:mm:ss')
-    format_date = workbook.add_format(num_format: 'yyyy-mm-dd')
-
-    format_footer = workbook.add_format
-    format_footer.set_italic
-    format_footer.set_color('gray')
-    format_footer.set_size(8)
-
-    worksheet.set_row(0, 18, header.count)
-
-    # Write a formatted and unformatted string, row and column notation.
-    worksheet.write_string(0, 0, title, format)
-
-    format_header = workbook.add_format  # Add a format
-    format_header.set_italic
-    format_header.set_bg_color('gray')
-    format_header.set_color('white')
-    count = 0
-    header.each do |item|
-      if item[:width]
-        worksheet.set_column(count, count, item[:width])
-      end
-      worksheet.write_string(2, count, item[:name], format_header)
-      count += 1
-    end
-
-    row_count = 2
-    result.each do |row|
-      row_count += 1
-      row_item_count = 0
-      row.each do |item|
-        if item.acts_like?(:time)
-          worksheet.write_date_time(row_count, row_item_count, time_in_localtime_for_excel(item, params[:timezone]), format_time) if item.present?
-        elsif item.acts_like?(:date)
-          worksheet.write_date_time(row_count, row_item_count, item.to_s, format_date) if item.present?
-        elsif item.is_a?(Integer) || item.is_a?(Float)
-          worksheet.write_number(row_count, row_item_count, item)
-        else
-          worksheet.write_string(row_count, row_item_count, item.to_s)
-        end
-        row_item_count += 1
-      end
-    end
-
-    row_count += 2
-    worksheet.write_string(row_count, 0, "#{Translation.translate(current_user.locale, 'Timezone')}: #{params[:timezone]}", format_footer)
-
-    workbook.close
-
-    # read file again
-    file = File.new(temp_file, 'r')
-    contents = file.read
-    file.close
-    contents
-  end
-
-  def time_in_localtime_for_excel(time, timezone)
-    return if time.blank?
-
-    if timezone.present?
-      offset = time.in_time_zone(timezone).utc_offset
-      time -= offset
-    end
-    time.utc.iso8601.to_s.sub(/Z$/, '')
-  end
 end
--- a/app/models/channel/email_parser.rb
+++ b/app/models/channel/email_parser.rb
@@ -420,7 +420,7 @@ returns
        end

        # only set value on _id if value/reference lookup exists
-        if mail[ header.to_sym ]
+        if mail[header.to_sym]

          Rails.logger.info "set_attributes_by_x_headers header #{header} found #{mail[header.to_sym]}"
          item_object.class.reflect_on_all_associations.map do |assoc|
@@ -429,14 +429,16 @@ returns

            Rails.logger.info "set_attributes_by_x_headers found #{assoc.class_name} lookup for '#{mail[header.to_sym]}'"
            item = assoc.class_name.constantize
-
            assoc_object = nil
-            if item.respond_to?(:name)
+            if item.new.respond_to?(:name)
              assoc_object = item.lookup(name: mail[header.to_sym])
            end
-            if !assoc_object && item.respond_to?(:login)
+            if !assoc_object && item.new.respond_to?(:login)
              assoc_object = item.lookup(login: mail[header.to_sym])
            end
+            if !assoc_object && item.new.respond_to?(:email)
+              assoc_object = item.lookup(email: mail[header.to_sym])
+            end

            if assoc_object.blank?

--- a/app/models/channel/filter/follow_up_check.rb
+++ b/app/models/channel/filter/follow_up_check.rb
@@ -29,9 +29,23 @@ module Channel::Filter::FollowUpCheck
    # get ticket# from attachment
    if setting.include?('attachment') && mail[:attachments]
      mail[:attachments].each do |attachment|
-        next if !attachment[:data]
+        next if attachment[:data].blank?
+        next if attachment[:preferences].blank?
+        next if attachment[:preferences]['Mime-Type'].blank?
+
+        if %r{text/html}i.match?(attachment[:preferences]['Mime-Type'])
+          begin
+            text = attachment[:data].html2text
+            ticket = Ticket::Number.check(text)
+          rescue => e
+            Rails.logger.error e
+          end
+        end
+
+        if %r{text/plain}i.match?(attachment[:preferences]['Mime-Type'])
+          ticket = Ticket::Number.check(attachment[:data])
+        end

-        ticket = Ticket::Number.check(attachment[:data].html2text)
        next if !ticket

        Rails.logger.debug { "Follow up for '##{ticket.number}' in attachment." }
--- a/app/models/concerns/has_search_index_backend.rb
+++ b/app/models/concerns/has_search_index_backend.rb
@@ -128,11 +128,12 @@ reload search index with full data
 =end

    def search_index_reload
-      tolerance       = 5
+      tolerance       = 10
      tolerance_count = 0
      ids = all.order(created_at: :desc).pluck(:id)
      ids.each do |item_id|
-        item = find(item_id)
+        item = find_by(id: item_id)
+        next if !item
        next if item.ignore_search_indexing?(:destroy)

        begin
@@ -140,6 +141,7 @@ reload search index with full data
        rescue => e
          logger.error "Unable to send #{item.class}.find(#{item.id}).search_index_update_backend backend: #{e.inspect}"
          tolerance_count += 1
+          sleep 15
          raise "Unable to send #{item.class}.find(#{item.id}).search_index_update_backend backend: #{e.inspect}" if tolerance_count == tolerance
        end
      end
--- a/app/models/object_manager/attribute.rb
+++ b/app/models/object_manager/attribute.rb
@@ -895,13 +895,13 @@ is certain attribute used by triggers, overviews or schedulers
  def check_name
    return if !name

-    raise 'Name can\'t get used, *_id and *_ids are not allowed' if name.match?(/_(id|ids)$/i) || name.match?(/^id$/i)
+    raise 'Name can\'t get used, *_id and *_ids are not allowed' if name.match?(/.+?_(id|ids)$/i)
    raise 'Spaces in name are not allowed' if name.match?(/\s/)
    raise 'Only letters from a-z, numbers from 0-9, and _ are allowed' if !name.match?(/^[a-z0-9_]+$/)
    raise 'At least one letters is needed' if !name.match?(/[a-z]/)

    # do not allow model method names as attributes
-    reserved_words = %w[destroy true false integer select drop create alter index table varchar blob date datetime timestamp]
+    reserved_words = %w[destroy true false integer select drop create alter index table varchar blob date datetime timestamp url icon initials avatar permission validate subscribe unsubscribe translate search _type _doc _id id]
    raise "#{name} is a reserved word, please choose a different one" if name.match?(/^(#{reserved_words.join('|')})$/)

    # fixes issue #2236 - Naming an attribute "attribute" causes ActiveRecord failure
--- a/app/models/ticket/article.rb
+++ b/app/models/ticket/article.rb
@@ -379,7 +379,7 @@ returns
    current_length = body.length
    return true if body.length <= limit

-    raise Exceptions::UnprocessableEntity, "body if article is to large, #{current_length} chars - only #{limit} allowed" if !ApplicationHandleInfo.postmaster?
+    raise Exceptions::UnprocessableEntity, "body of article is too large, #{current_length} chars - only #{limit} allowed" if !ApplicationHandleInfo.postmaster?

    logger.warn "WARNING: cut string because of database length #{self.class}.body(#{limit} but is #{current_length})"
    self.body = body[0, limit]
--- a/app/models/ticket/subject.rb
+++ b/app/models/ticket/subject.rb
@@ -50,6 +50,8 @@ returns
 =end

  def subject_clean(subject)
+    return '' if subject.blank?
+
    ticket_hook         = Setting.get('ticket_hook')
    ticket_hook_divider = Setting.get('ticket_hook_divider')
    ticket_subject_size = Setting.get('ticket_subject_size')
--- a/app/views/mailer/ticket_create/pl.html.erb
+++ b/app/views/mailer/ticket_create/pl.html.erb
@@ -2,7 +2,7 @@ Nowe zgłoszenie (#{ticket.title})

 <div>Witaj #{recipient.firstname},</div>
 <br>
-<divOtrzymałeś nowe zgłoszenie (#{ticket.title}) od "<b>#{current_user.longname}</b>".</div>
+<div>Otrzymałeś nowe zgłoszenie (#{ticket.title}) od "<b>#{current_user.longname}</b>".</div>
 <br>
 <div>
 #{t('Group')}: #{ticket.group.name}<br>
--- a/app/views/mailer/ticket_escalation/pl.html.erb
+++ b/app/views/mailer/ticket_escalation/pl.html.erb
@@ -15,4 +15,4 @@ Zgłoszenie (#{ticket.title}) zostało eskalowane
 <br>
 <div>
  <a href="#{config.http_type}://#{config.fqdn}/#ticket/zoom/#{ticket.id}" target="zammad_app">#{t('Zobacz w Zammad')}</a>
-</div
+</div>
--- a/app/views/tests/model_ticket.html.erb
+++ b/app/views/tests/model_ticket.html.erb
@@ -0,0 +1,16 @@
+
+<link rel="stylesheet" href="/assets/tests/qunit-1.21.0.css">
+<script src="/assets/tests/qunit-1.21.0.js"></script>
+<script src="/assets/tests/model_ticket.js"></script>
+
+<style type="text/css">
+body {
+  padding-top: 0px;
+}
+</style>
+
+<script type="text/javascript">
+</script>
+
+<div id="qunit" class="u-dontfold"></div>
+
--- a/config/application.rb
+++ b/config/application.rb
@@ -1,6 +1,7 @@
 require_relative 'boot'

 require 'rails/all'
+require_relative 'issue_2656_workaround_for_rails_issue_33600'

 # DO NOT REMOVE THIS LINE - see issue #2037
 Bundler.setup
--- a/config/issue_2656_workaround_for_rails_issue_33600.rb
+++ b/config/issue_2656_workaround_for_rails_issue_33600.rb
@@ -0,0 +1,22 @@
+# This temporary workaround for issue #2656.
+# The root cause is an issue in Rails: https://github.com/rails/rails/issues/33600
+# It disables database connnection reaping by setting `reaping_frequency` to 0 for each environment in the config/database.yml file.
+# It restores the DB connection reaping behavior Rails > 5.2 had.
+# It was proposed in a comment on the Rails issue: https://github.com/rails/rails/issues/33600#issuecomment-415395901
+# It was confirmed by @matthewd (a Rails core maintainer) in another comment: https://github.com/rails/rails/issues/33600#issuecomment-415400522
+module Rails
+  class Application
+    class Configuration < ::Rails::Engine::Configuration
+
+      alias database_configuration_original database_configuration
+
+      def database_configuration
+        database_configuration_original&.transform_values do |config|
+          config.merge(
+            'reaping_frequency' => 0
+          )
+        end
+      end
+    end
+  end
+end
--- a/config/routes/test.rb
+++ b/config/routes/test.rb
@@ -6,6 +6,7 @@ Zammad::Application.routes.draw do
  match '/tests_model',                       to: 'tests#model',                      via: :get
  match '/tests_model_binding',               to: 'tests#model_binding',              via: :get
  match '/tests_model_ui',                    to: 'tests#model_ui',                   via: :get
+  match '/tests_model_ticket',                to: 'tests#model_ticket',               via: :get
  match '/tests_form',                        to: 'tests#form',                       via: :get
  match '/tests_form_tree_select',            to: 'tests#form_tree_select',           via: :get
  match '/tests_form_find',                   to: 'tests#form_find',                  via: :get
--- a/contrib/packager.io/functions
+++ b/contrib/packager.io/functions
@@ -303,6 +303,11 @@ function set_env_vars () {
  zammad config:set RUBY_GC_MALLOC_LIMIT_MAX=${ZAMMAD_RUBY_GC_MALLOC_LIMIT_MAX:=2177216}
  zammad config:set RUBY_GC_OLDMALLOC_LIMIT=${ZAMMAD_RUBY_GC_OLDMALLOC_LIMIT:=2177216}
  zammad config:set RUBY_GC_OLDMALLOC_LIMIT_MAX=${ZAMMAD_RUBY_GC_OLDMALLOC_LIMIT_MAX:=3000100}
+  if [[ "$(zammad config:get RAILS_LOG_TO_STDOUT)" == "enabled" ]];then
+    echo 'Setting default Logging to file, set via "zammad config:set RAILS_LOG_TO_STDOUT=true" if you want to log to STDOUT!'
+    zammad config:set RAILS_LOG_TO_STDOUT=
+  fi
+
 }

 function final_message () {
--- a/db/migrate/20190408000001_issue_2541_fix_notification_email_without_body.rb
+++ b/db/migrate/20190408000001_issue_2541_fix_notification_email_without_body.rb
@@ -4,6 +4,10 @@ class Issue2541FixNotificationEmailWithoutBody < ActiveRecord::Migration[5.1]
    # return if it's a new setup
    return if !Setting.find_by(name: 'system_init_done')

+    # there might be Job/Trigger selectors referencing the current user
+    # that get e.g. validated in callbacks
+    UserInfo.current_user_id = 1
+
    # update jobs and triggers
    [::Job, ::Trigger].each do |model|
      model.where(active: true).each do |record|
--- a/lib/core_ext/string.rb
+++ b/lib/core_ext/string.rb
@@ -284,6 +284,7 @@ class String
        chr_orig
      end
    end
+    string = string.utf8_encode(fallback: :read_as_sanitized_binary)

    # remove tailing empty spaces
    string.gsub!(/[[:blank:]]+$/, '')
--- a/lib/excel_sheet.rb
+++ b/lib/excel_sheet.rb
@@ -0,0 +1,172 @@
+class ExcelSheet
+
+  def initialize(title:, header:, records:, timezone: nil, locale:)
+    @title           = title
+    @header          = header
+    @records         = records
+    @timezone        = timezone.presence || Setting.get('timezone_default')
+    @locale          = locale || 'en-en'
+    @tempfile        = Tempfile.new('excel-export.xls')
+    @workbook        = WriteExcel.new(@tempfile)
+    @worksheet       = @workbook.add_worksheet
+    @contents        = nil
+    @current_row     = 0
+    @current_column  = 0
+
+    @lookup_cache = {}
+
+    @format_time = @workbook.add_format(num_format: 'yyyy-mm-dd hh:mm:ss')
+    @format_date = @workbook.add_format(num_format: 'yyyy-mm-dd')
+
+    @format_headline = @workbook.add_format
+    @format_headline.set_bold
+    @format_headline.set_size(14)
+    @format_headline.set_color('black')
+
+    @format_header = @workbook.add_format
+    @format_header.set_italic
+    @format_header.set_bg_color('gray')
+    @format_header.set_color('white')
+
+    @format_footer = @workbook.add_format
+    @format_footer.set_italic
+    @format_footer.set_color('gray')
+    @format_footer.set_size(8)
+  end
+
+  def contents
+    file = File.new(@tempfile, 'r')
+    contents = file.read
+    file.close
+    contents
+  end
+
+  def content
+    gen_header
+    gen_rows
+    gen_footer
+    contents
+  end
+
+  def gen_header
+    @worksheet.write_string(@current_row, @current_column, @title, @format_headline)
+    @worksheet.set_row(0, 18)
+
+    @current_row += 2
+    @current_column = 0
+    @header.each do |header|
+      if header[:width]
+        @worksheet.set_column(@current_column, @current_column, header[:width])
+      end
+      @worksheet.write_string(@current_row, @current_column, header[:display] || header[:name], @format_header)
+      @current_column += 1
+    end
+  end
+
+  def gen_rows
+    @records.each do |record|
+      gen_row_by_array(record)
+    end
+  end
+
+  def gen_row_by_array(record)
+    @current_row += 1
+    @current_column = 0
+    record.each do |item|
+      begin
+        if item.acts_like?(:time)
+          value_convert(item, nil, { data_type: 'datetime' })
+        elsif item.acts_like?(:date)
+          value_convert(item, nil, { data_type: 'datetime' })
+        elsif item.is_a?(Integer) || item.is_a?(Float)
+          value_convert(item, nil, { data_type: 'integer' })
+        else
+          value_convert(item, nil, { data_type: 'string' })
+        end
+      rescue => e
+        Rails.logger.error e
+      end
+      @current_column += 1
+    end
+  end
+
+  def gen_row_by_header(record, additional = {})
+    @current_row += 1
+    @current_column = 0
+    @header.each do |header|
+      begin
+        value_convert(record, header[:name], header, additional)
+      rescue => e
+        Rails.logger.error e
+      end
+      @current_column += 1
+    end
+  end
+
+  def gen_footer
+    @current_row += 2
+    @worksheet.write_string(@current_row, 0, "#{Translation.translate(@locale, 'Timezone')}: #{@timezone}", @format_footer)
+    @workbook.close
+  end
+
+  def timestamp_in_localtime(time)
+    return if time.blank?
+
+    time.in_time_zone(@timezone).strftime('%F %T') # "2019-08-19 16:21:52"
+  end
+
+  def value_lookup(record, attribute, additional)
+    value = record[attribute.to_sym]
+    if attribute[-3, 3] == '_id'
+      ref = attribute[0, attribute.length - 3]
+      if record.respond_to?(ref.to_sym)
+        @lookup_cache[attribute] ||= {}
+        return @lookup_cache[attribute][value] if @lookup_cache[attribute][value]
+
+        ref_object = record.send(ref.to_sym)
+        ref_name = value
+        if ref_object.respond_to?(:fullname)
+          ref_name = ref_object.fullname
+        elsif ref_object.respond_to?(:name)
+          ref_name = ref_object.name
+        end
+        @lookup_cache[attribute][value] = ref_name
+        return ref_name
+      end
+    end
+    value = record.try(attribute)
+
+    # if no value exists, check additional values
+    if !value && additional && additional[attribute.to_sym]
+      value = additional[attribute.to_sym]
+    end
+    if value.is_a?(Array)
+      value = value.join(',')
+    end
+    value
+  end
+
+  def value_convert(record, attribute, object, additional = {})
+    value = if attribute
+              value_lookup(record, attribute, additional)
+            else
+              record
+            end
+    case object[:data_type]
+    when 'boolean', 'select'
+      if object[:data_option] && object[:data_option]['options'] && object[:data_option]['options'][value]
+        value = object[:data_option]['options'][value]
+      end
+      @worksheet.write_string(@current_row, @current_column, value) if value.present?
+    when 'datetime'
+      @worksheet.write_date_time(@current_row, @current_column, timestamp_in_localtime(value), @format_time) if value.present?
+    when 'date'
+      @worksheet.write_date_time(@current_row, @current_column, value.to_s, @format_date) if value.present?
+    when 'integer'
+      @worksheet.write_number(@current_row, @current_column, value) if value.present?
+    else
+      @worksheet.write_string(@current_row, @current_column, value.to_s) if value.present?
+    end
+  end
+
+end
--- a/lib/excel_sheet/ticket.rb
+++ b/lib/excel_sheet/ticket.rb
@@ -0,0 +1,103 @@
+class ExcelSheet::Ticket < ExcelSheet
+
+=begin
+
+  excel = ExcelSheet::Ticket.new(
+    title:                        "#{year}-#{month}",
+    ticket_ids:                   ticket_ids,
+    additional_attributes:        additional_attributes,
+    additional_attributes_header: additional_attributes_header,
+    timezone:                     params[:timezone],
+    locale:                       current_user.locale,
+  )
+
+  excel.content
+
+=end
+
+  def initialize(params)
+    @ticket_ids                   = params[:ticket_ids] || []
+    @additional_attributes        = params[:additional_attributes] || []
+    @additional_attributes_header = params[:additional_attributes_header] || []
+
+    super(
+      title:    params[:title],
+      header:   ticket_header,
+      records:  [],
+      timezone: params[:timezone],
+      locale:   params[:locale]
+    )
+  end
+
+  def ticket_header
+    header = [
+      { display: '#', name: 'number', width: 18, data_type: 'string' },
+      { display: 'Title', name: 'title', width: 34, data_type: 'string' },
+      { display: 'State', name: 'state_id', width: 14, data_type: 'string' },
+      { display: 'Priority', name: 'priority_id', width: 14, data_type: 'string' },
+      { display: 'Group', name: 'group_id', width: 20, data_type: 'string' },
+      { display: 'Owner', name: 'owner_id', width: 20, data_type: 'string' },
+      { display: 'Customer', name: 'customer_id', width: 20, data_type: 'string' },
+      { display: 'Organization', name: 'organization_id', width: 20, data_type: 'string' },
+      { display: 'Create Channel', name: 'create_article_type_id', width: 10, data_type: 'string' },
+      { display: 'Sender', name: 'create_article_sender_id', width: 14, data_type: 'string' },
+      { display: 'Tags', name: 'tag_list', width: 20, data_type: 'string' },
+      { display: 'Time Units Total', name: 'time_unit', width: 10, data_type: 'float' },
+    ]
+
+    header = header.concat(@additional_attributes_header) if @additional_attributes_header
+
+    # ObjectManager attributes
+    objects = ObjectManager::Attribute.where(active:           true,
+                                             to_create:        false,
+                                             object_lookup_id: ObjectLookup.lookup(name: 'Ticket').id)
+                                      .pluck(:name, :display, :data_type, :data_option)
+                                      .map { |name, display, data_type, data_option| { name: name, display: display, data_type: data_type, data_option: data_option, width: 20 } }
+    objects.each do |object|
+      already_exists = false
+      header.each do |local_header|
+        next if local_header[:name] != object[:name]
+
+        already_exists = true
+        break
+      end
+      next if already_exists
+
+      header.push object
+    end
+
+    header = header.concat([
+                             { display: 'Created At', name: 'created_at', width: 18, data_type: 'datetime' },
+                             { display: 'Updated At', name: 'updated_at', width: 18, data_type: 'datetime' },
+                             { display: 'Closed At', name: 'close_at', width: 18, data_type: 'datetime' },
+                             { display: 'Close Escalation At', name: 'close_escalation_at', width: 18, data_type: 'datetime' },
+                             { display: 'Close In Min', name: 'close_in_min', width: 10, data_type: 'integer' },
+                             { display: 'Close Diff In Min', name: 'close_diff_in_min', width: 10, data_type: 'integer' },
+                             { display: 'First Response At', name: 'first_response_at', width: 18, data_type: 'datetime' },
+                             { display: 'First Response Escalation At', name: 'first_response_escalation_at', width: 18, data_type: 'datetime' },
+                             { display: 'First Response In Min', name: 'first_response_in_min', width: 10, data_type: 'integer' },
+                             { display: 'First Response Diff In Min', name: 'first_response_diff_in_min', width: 10, data_type: 'integer' },
+                             { display: 'Update Escalation At', name: 'update_escalation_at', width: 18, data_type: 'datetime' },
+                             { display: 'Update In Min', name: 'update_in_min', width: 10, data_type: 'integer' },
+                             { display: 'Update Diff In Min', name: 'update_diff_in_min', width: 10, data_type: 'integer' },
+                             { display: 'Last Contact At', name: 'last_contact_at', width: 18, data_type: 'datetime' },
+                             { display: 'Last Contact Agent At', name: 'last_contact_agent_at', width: 18, data_type: 'datetime' },
+                             { display: 'Last Contact Customer At', name: 'last_contact_customer_at', width: 18, data_type: 'datetime' },
+                             { display: 'Article Count', name: 'article_count', width: 10, data_type: 'integer' },
+                             { display: 'Escalation At', name: 'escalation_at', width: 18, data_type: 'datetime' },
+                           ])
+    header
+  end
+
+  def gen_rows
+    @ticket_ids.each_with_index do |ticket_id, index|
+      ticket = ::Ticket.lookup(id: ticket_id)
+      raise "Can't find Ticket with ID #{ticket_id} for '#{@title}' #{self.class.name} generation" if !ticket
+
+      gen_row_by_header(ticket, @additional_attributes[index])
+    rescue => e
+      Rails.logger.error e
+    end
+  end
+
+end
--- a/lib/html_sanitizer.rb
+++ b/lib/html_sanitizer.rb
@@ -1,6 +1,6 @@
 class HtmlSanitizer
  LINKABLE_URL_SCHEMES = URI.scheme_list.keys.map(&:downcase) - ['mailto'] + ['tel']
-  PROCESSING_TIMEOUT = 10
+  PROCESSING_TIMEOUT = 20
  UNPROCESSABLE_HTML_MSG = 'This message cannot be displayed due to HTML processing issues. Download the raw message below and open it via an Email client if you still wish to view it.'.freeze

 =begin
@@ -205,6 +205,7 @@ satinize html string based on whiltelist
      Loofah.fragment(string).scrub!(scrubber_link).to_s
    end
  rescue Timeout::Error
+    Rails.logger.error "Could not process string via HtmlSanitizer.strict in #{PROCESSING_TIMEOUT} seconds. Current state: #{string}"
    UNPROCESSABLE_HTML_MSG
  end

@@ -237,6 +238,7 @@ cleanup html string:
      string
    end
  rescue Timeout::Error
+    Rails.logger.error "Could not process string via HtmlSanitizer.cleanup in #{PROCESSING_TIMEOUT} seconds. Current state: #{string}"
    UNPROCESSABLE_HTML_MSG
  end

--- a/lib/import/otrs/user.rb
+++ b/lib/import/otrs/user.rb
@@ -153,6 +153,7 @@ module Import
        return result if role_object['GroupIDs'].blank?

        permissions = role_object['GroupIDs'][ group['ID'] ]
+        permissions ||= role_object['GroupIDs'][ group['ID'].to_s ]

        return result if !permissions

--- a/lib/search_index_backend.rb
+++ b/lib/search_index_backend.rb
@@ -21,7 +21,7 @@ info about used search index machine
      {
        json:         true,
        open_timeout: 8,
-        read_timeout: 12,
+        read_timeout: 14,
        user:         Setting.get('es_user'),
        password:     Setting.get('es_password'),
      }
@@ -84,7 +84,7 @@ update processors
            {
              json:         true,
              open_timeout: 8,
-              read_timeout: 12,
+              read_timeout: 60,
              user:         Setting.get('es_user'),
              password:     Setting.get('es_password'),
            }
@@ -108,7 +108,7 @@ update processors
          {
            json:         true,
            open_timeout: 8,
-            read_timeout: 12,
+            read_timeout: 60,
            user:         Setting.get('es_user'),
            password:     Setting.get('es_password'),
          }
@@ -181,7 +181,7 @@ create/update/delete index
      {
        json:         true,
        open_timeout: 8,
-        read_timeout: 30,
+        read_timeout: 60,
        user:         Setting.get('es_user'),
        password:     Setting.get('es_password'),
      }
@@ -219,7 +219,7 @@ add new object to search index
      {
        json:         true,
        open_timeout: 8,
-        read_timeout: 16,
+        read_timeout: 60,
        user:         Setting.get('es_user'),
        password:     Setting.get('es_password'),
      }
@@ -246,7 +246,12 @@ remove whole data from index
 =end

  def self.remove(type, o_id = nil)
-    url = build_url(type, o_id, false, false)
+    url = if o_id
+            build_url(type, o_id, false, true)
+          else
+            build_url(type, o_id, false, false)
+          end
+
    return if url.blank?

    Rails.logger.info "# curl -X DELETE \"#{url}\""
@@ -255,7 +260,7 @@ remove whole data from index
      url,
      {
        open_timeout: 8,
-        read_timeout: 16,
+        read_timeout: 60,
        user:         Setting.get('es_user'),
        password:     Setting.get('es_password'),
      }
@@ -285,10 +290,12 @@ remove whole data from index

  result = SearchIndexBackend.search('search query', ['User', 'Organization'], limit: limit)

-  result = SearchIndexBackend.search('search query', 'User', limit: limit)
+- result = SearchIndexBackend.search('search query', 'User', limit: limit)

  result = SearchIndexBackend.search('search query', 'User', limit: limit, sort_by: ['updated_at'], order_by: ['desc'])

+  result = SearchIndexBackend.search('search query', 'User', limit: limit, sort_by: ['active', updated_at'], order_by: ['desc', 'desc'])
+
  result = [
    {
      :id   => 123,
@@ -727,7 +734,25 @@ return true if backend is configured
    "#{local_index}_#{index.underscore.tr('/', '_')}"
  end

-  def self.build_url(type = nil, o_id = nil, pipeline = true, with_type = true)
+=begin
+
+generate url for index or document access (only for internal use)
+
+  # url to access single document in index (in case with_pipeline or not)
+  url = SearchIndexBackend.build_url('User', 123, with_pipeline)
+
+  # url to access whole index
+  url = SearchIndexBackend.build_url('User')
+
+  # url to access document definition in index (only es6 and higher)
+  url = SearchIndexBackend.build_url('User', nil, false, true)
+
+  # base url
+  url = SearchIndexBackend.build_url
+
+=end
+
+  def self.build_url(type = nil, o_id = nil, with_pipeline = true, with_document_type = true)
    return if !SearchIndexBackend.enabled?

    # for elasticsearch 5.6 and lower
@@ -735,7 +760,7 @@ return true if backend is configured
    if Setting.get('es_multi_index') == false
      url = Setting.get('es_url')
      url = if type
-              if pipeline == true
+              if with_pipeline == true
                url_pipline = Setting.get('es_pipeline')
                if url_pipline.present?
                  url_pipline = "?pipeline=#{url_pipline}"
@@ -754,7 +779,7 @@ return true if backend is configured

    # for elasticsearch 6.x and higher
    url = Setting.get('es_url')
-    if pipeline == true
+    if with_pipeline == true
      url_pipline = Setting.get('es_pipeline')
      if url_pipline.present?
        url_pipline = "?pipeline=#{url_pipline}"
@@ -762,20 +787,36 @@ return true if backend is configured
    end
    if type
      index = build_index_name(type)
-      if with_type == false
+
+      # access (e. g. creating or dropping) whole index
+      if with_document_type == false
        return "#{url}/#{index}"
      end

+      # access single document in index (e. g. drop or add document)
      if o_id
        return "#{url}/#{index}/_doc/#{o_id}#{url_pipline}"
      end

+      # access document type (e. g. creating or dropping document mapping)
      return "#{url}/#{index}/_doc#{url_pipline}"
    end
    "#{url}/"
  end

-  def self.build_search_url(index)
+=begin
+
+generate url searchaccess (only for internal use)
+
+  # url search access with single index
+  url = SearchIndexBackend.build_search_url('User')
+
+  # url to access all over es
+  url = SearchIndexBackend.build_search_url
+
+=end
+
+  def self.build_search_url(index = nil)

    # for elasticsearch 5.6 and lower
    if Setting.get('es_multi_index') == false
--- a/lib/sessions/backend/ticket_overview_list.rb
+++ b/lib/sessions/backend/ticket_overview_list.rb
@@ -98,9 +98,9 @@ class Sessions::Backend::TicketOverviewList < Sessions::Backend::Base
    index_and_lists.each do |data|

      # do not deliver unchanged lists
-      next if @last_overview[data[:overview][:id]] == data
+      next if @last_overview[data[:overview][:id]] == [data[:tickets], data[:overview]]

-      @last_overview[data[:overview][:id]] = data
+      @last_overview[data[:overview][:id]] = [data[:tickets], data[:overview]]

      assets = {}
      overview = Overview.lookup(id: data[:overview][:id])
--- a/lib/user_info.rb
+++ b/lib/user_info.rb
@@ -14,9 +14,7 @@ module UserInfo
    end

    yield
-
-    return if !reset_current_user_id
-
-    UserInfo.current_user_id = nil
+  ensure
+    UserInfo.current_user_id = nil if reset_current_user_id
  end
 end
--- a/lib/zammad/application/initializer/db_preflight_check.rb
+++ b/lib/zammad/application/initializer/db_preflight_check.rb
@@ -1,6 +1,7 @@
 require 'zammad/application/initializer/db_preflight_check/base'
 require 'zammad/application/initializer/db_preflight_check/mysql2'
 require 'zammad/application/initializer/db_preflight_check/postgresql'
+require 'zammad/application/initializer/db_preflight_check/nulldb'

 module Zammad
  class Application
--- a/lib/zammad/application/initializer/db_preflight_check/nulldb.rb
+++ b/lib/zammad/application/initializer/db_preflight_check/nulldb.rb
@@ -0,0 +1,12 @@
+module Zammad
+  class Application
+    class Initializer
+      module DBPreflightCheck
+        module Nulldb
+          # no-op
+          def self.perform; end
+        end
+      end
+    end
+  end
+end
--- a/public/assets/tests/model_ticket.js
+++ b/public/assets/tests/model_ticket.js
@@ -0,0 +1,104 @@
+window.onload = function() {
+
+  App.Ticket.refresh([{
+    id: 1,
+    title: 'ticket1',
+    state_id: 1,
+    customer_id: 33,
+    organization_id: 1,
+    owner_id: 1,
+  },
+  {
+    id: 2,
+    title: 'ticket2',
+    state_id: 1,
+    customer_id: 44,
+    organization_id: 1,
+    owner_id: 1,
+  },
+  {
+    id: 3,
+    title: 'ticket3',
+    state_id: 1,
+    customer_id: 55,
+    organization_id: undefined,
+    owner_id: 1,
+  },
+  {
+    id: 4,
+    title: 'ticket4',
+    state_id: 1,
+    customer_id: 66,
+    organization_id: undefined,
+    owner_id: 1,
+    group_id: 1,
+  }])
+
+  App.User.refresh([{
+    id: 33,
+    login: 'hh@1example.com',
+    firstname: 'Harald',
+    lastname: 'Habebe',
+    email: 'hh1@example.com',
+    organization_id: 1,
+    role_ids: [3],
+    active: true,
+  },
+  {
+    id: 44,
+    login: 'hh2@example.com',
+    firstname: 'Harald',
+    lastname: 'Habebe',
+    email: 'hh2@example.com',
+    organization_id: 2,
+    role_ids: [3],
+    active: true,
+  },
+  {
+    id: 55,
+    login: 'hh3example.com',
+    firstname: 'Harald',
+    lastname: 'Habebe',
+    email: 'hh3@example.com',
+    organization_id: undefined,
+    role_ids: [3],
+    active: true,
+  }])
+
+  test('ticket.editabe customer user #1', function() {
+    App.Session.set(33)
+    ticket1 = App.Ticket.find(1);
+    ok(ticket1.editable(), 'access via customer_id');
+    ticket2 = App.Ticket.find(2);
+    ok(ticket2.editable(), 'access via organization_id');
+    ticket3 = App.Ticket.find(3);
+    ok(!ticket3.editable(), 'no access');
+    ticket4 = App.Ticket.find(4);
+    ok(!ticket4.editable(), 'no access');
+  });
+
+  test('ticket.editabe customer user #2', function() {
+    App.Session.set(44)
+    ticket1 = App.Ticket.find(1);
+    ok(!ticket1.editable(), 'no access');
+    ticket2 = App.Ticket.find(2);
+    ok(ticket2.editable(), 'access via customer_id');
+    ticket3 = App.Ticket.find(3);
+    ok(!ticket3.editable(), 'no access');
+    ticket4 = App.Ticket.find(4);
+    ok(!ticket4.editable(), 'no access');
+  });
+
+  test('ticket.editabe customer user #3', function() {
+    App.Session.set(55)
+    ticket1 = App.Ticket.find(1);
+    ok(!ticket1.editable(), 'no access');
+    ticket2 = App.Ticket.find(2);
+    ok(!ticket2.editable(), 'no access');
+    ticket3 = App.Ticket.find(3);
+    ok(ticket3.editable(), 'access via customer_id');
+    ticket4 = App.Ticket.find(4);
+    ok(!ticket4.editable(), 'no access');
+  });
+
+}
--- a/spec/db/migrate/issue_2541_fix_notification_email_without_body_spec.rb
+++ b/spec/db/migrate/issue_2541_fix_notification_email_without_body_spec.rb
@@ -44,7 +44,23 @@ RSpec.describe Issue2541FixNotificationEmailWithoutBody, type: :db_migration do
      it "updates empty perform['notification.email']['body'] attribute" do
        expect { migrate }.to change { job.reload.perform['notification.email']['body'] }.from('').to('-')
      end
+
+      context 'when selector contains current_user.id' do
+        subject(:job) do
+          UserInfo.ensure_current_user_id do
+
+            create(:job, condition: { 'ticket.owner_id' => { 'operator' => 'is', 'pre_condition' => 'current_user.id', 'value' => '', 'value_completion' => '' } } )
+          end
+        end
+
+        let(:type) { 'notification.email' }
+
+        it "updates empty perform['notification.email']['body'] attribute" do
+          expect { migrate }.to change { job.reload.perform['notification.email']['body'] }.from('').to('-')
+        end
+      end
    end
+
  end

  describe 'scheduler management' do
--- a/spec/lib/core_ext/string_spec.rb
+++ b/spec/lib/core_ext/string_spec.rb
@@ -542,6 +542,18 @@ RSpec.describe String do
      TEXT
    end

+    context 'html encoding' do
+      it 'converts &Auml; in Ä' do
+        expect('<div>test something.&Auml;</div>'.html2text)
+          .to eq('test something.Ä')
+      end
+
+      it 'strips invalid html encoding chars' do
+        expect('<div>test something.&#55357;</div>'.html2text)
+          .to eq('test something.í ˝')
+      end
+    end
+
    context 'performance tests' do
      let(:filler) do
        %(<p>some word <a href="http://example.com?domain?example.com">some url</a> and the end.</p>\n) * 11 + "\n"
@@ -574,6 +586,7 @@ RSpec.describe String do
          </html>
        HTML
      end
+
    end
  end

--- a/spec/lib/excel_sheet_spec.rb
+++ b/spec/lib/excel_sheet_spec.rb
@@ -0,0 +1,14 @@
+require 'rails_helper'
+
+RSpec.describe ExcelSheet do
+
+  describe '.timestamp_in_localtime' do
+
+    let(:document) { described_class.new(title: 'some title', header: [], records: [], timezone: 'Europe/Berlin', locale: 'de-de') }
+
+    it 'does convert UTC timestamp to local system based timestamp' do
+      expect(document.timestamp_in_localtime(Time.parse('2019-08-08T01:00:05Z').in_time_zone)).to eq('2019-08-08 03:00:05')
+    end
+
+  end
+end
--- a/spec/lib/search_index_backend_spec.rb
+++ b/spec/lib/search_index_backend_spec.rb
@@ -18,9 +18,27 @@ RSpec.describe SearchIndexBackend, searchindex: true do
  end

  describe '.search' do
-    subject(:search) { described_class.search(query, index, limit: 3000) }
+
+    context 'query finds results' do
+
+      let(:record_type) { 'Ticket'.freeze }
+      let(:record) { create :ticket }
+
+      before do
+        described_class.add(record_type, record)
+        # give ES time to update the index
+        sleep 2
+      end
+
+      it 'finds added records' do
+        result = described_class.search(record.number, record_type, sort_by: ['updated_at'], order_by: ['desc'])
+        expect(result).to eq([{ id: record.id.to_s, type: record_type }])
+      end
+    end

    context 'for query with no results' do
+      subject(:search) { described_class.search(query, index, limit: 3000) }
+
      let(:query) { 'preferences.notification_sound.enabled:*' }

      context 'on a single index' do
@@ -107,26 +125,45 @@ RSpec.describe SearchIndexBackend, searchindex: true do
  end

  describe '.remove' do
-    context 'ticket' do
-      it 'from index after ticket delete' do
+    context 'record gets deleted' do

-        skip('No ES configured') if !SearchIndexBackend.enabled?
+      let(:record_type) { 'Ticket'.freeze }
+      let(:deleted_record) { create :ticket }

-        ticket = create :ticket
-        described_class.add('Ticket', ticket)
-
-        # give es time to rebuild index
+      before do
+        described_class.add(record_type, deleted_record)
+        # give ES time to update the index
        sleep 2
-        result = described_class.search(ticket.number, 'Ticket', sort_by: ['updated_at'], order_by: ['desc'])
-        expect(result).to eq([{ id: ticket.id.to_s, type: 'Ticket' }])
+      end

-        described_class.remove('Ticket', ticket.id)
-        # give es time to rebuild index
+      it 'removes record from search index' do
+        described_class.remove(record_type, deleted_record.id)
+        # give ES time to update the index
        sleep 2

-        result = described_class.search(ticket.number, 'Ticket', sort_by: ['updated_at'], order_by: ['desc'])
+        result = described_class.search(deleted_record.number, record_type, sort_by: ['updated_at'], order_by: ['desc'])
        expect(result).to eq([])
      end
+
+      context 'other records present' do
+
+        let(:other_record) { create :ticket }
+
+        before do
+          described_class.add(record_type, other_record)
+          # give ES time to update the index
+          sleep 2
+        end
+
+        it "doesn't remove other records" do
+          described_class.remove(record_type, deleted_record.id)
+          # give ES time to update the index
+          sleep 2
+
+          result = described_class.search(other_record.number, record_type, sort_by: ['updated_at'], order_by: ['desc'])
+          expect(result).to eq([{ id: other_record.id.to_s, type: record_type }])
+        end
+      end
    end
  end
 end
--- a/spec/lib/sessions/backend/ticket_overview_list_spec.rb
+++ b/spec/lib/sessions/backend/ticket_overview_list_spec.rb
@@ -62,8 +62,8 @@ RSpec.describe Sessions::Backend::TicketOverviewList do
      end

      context 'when called twice, after changes have occurred to the Ticket table' do
+        let!(:ticket) { create(:ticket, group: group) }
        let!(:first_call) { collection.push }
-        let!(:ticket) { create(:ticket, group: group, owner: admin) }

        context 'before the TTL has passed' do
          it 'returns nil' do
@@ -73,16 +73,8 @@ RSpec.describe Sessions::Backend::TicketOverviewList do
          context 'after .reset with the user’s id' do
            before { described_class.reset(admin.id) }

-            it 'returns an updated set of results' do
-              expect(collection.push)
-                .to be_an(Array)
-                .and have_attributes(length: Ticket::Overviews.all(current_user: admin).count)
-                .and all(match({ event: 'ticket_overview_list', data: hash_including(:assets) }))
-            end
-
-            it 'includes FE assets for the new ticket' do
-              expect(collection.push.first[:data][:assets])  # first overview is "My assigned tickets",
-                .to eq(ticket.assets({}))                    # and newly created ticket was assigned to admin
+            it 'returns nil because no ticket and no overview has changed' do
+              expect(collection.push).to be nil
            end
          end
        end
@@ -90,32 +82,16 @@ RSpec.describe Sessions::Backend::TicketOverviewList do
        context 'after the TTL has passed' do
          before { travel(ttl + 1) }

-          it 'returns an updated set of results' do
-            expect(collection.push)
-              .to be_an(Array)
-              .and have_attributes(length: Ticket::Overviews.all(current_user: admin).count)
-              .and all(match({ event: 'ticket_overview_list', data: hash_including(:assets) }))
-          end
-
-          it 'includes FE assets for the new ticket' do
-            expect(collection.push.first[:data][:assets])  # first overview is "My assigned tickets",
-              .to eq(ticket.assets({}))                    # and newly created ticket was assigned to admin
+          it 'returns an empty result' do
+            expect(collection.push).to eq nil
          end
        end

        context 'after two hours have passed' do
          before { travel(2.hours + 1.second) }

-          it 'returns an updated set of results' do
-            expect(collection.push)
-              .to be_an(Array)
-              .and have_attributes(length: Ticket::Overviews.all(current_user: admin).count)
-              .and all(match({ event: 'ticket_overview_list', data: hash_including(:assets) }))
-          end
-
-          it 'includes FE assets for old AND new tickets/overviews' do
-            expect(collection.push.first[:data][:assets])       # first overview is "My assigned tickets",
-              .to eq(Overview.first.assets(ticket.assets({})))  # and newly created ticket was assigned to admin
+          it 'returns an empty result' do
+            expect(collection.push).to eq nil
          end
        end
      end
@@ -136,40 +112,21 @@ RSpec.describe Sessions::Backend::TicketOverviewList do
            it 'returns an updated set of results' do
              expect(collection.push)
                .to be_an(Array)
-                .and have_attributes(length: Ticket::Overviews.all(current_user: admin).count)
+                .and have_attributes(length: 1)
                .and all(match({ event: 'ticket_overview_list', data: hash_including(:assets) }))
            end
-
-            it 'includes FE assets for the touched overview' do
-              expect(collection.push.first[:data][:assets])
-                .to eq(Overview.first.assets({}))
-            end
-          end
-        end
-
-        context 'after the TTL has passed' do
-          before { travel(ttl + 1) }
-
-          it 'includes FE assets for the touched overview' do
-            expect(collection.push.first[:data][:assets])
-              .to eq(Overview.first.assets({}))
          end
        end

        context 'after two hours have passed' do
          before { travel(2.hours + 1.second) }

-          it 'returns an updated set of results' do
+          it 'returns an empty result' do
            expect(collection.push)
              .to be_an(Array)
-              .and have_attributes(length: Ticket::Overviews.all(current_user: admin).count)
+              .and have_attributes(length: 1)
              .and all(match({ event: 'ticket_overview_list', data: hash_including(:assets) }))
          end
-
-          it 'includes FE assets for old AND new tickets/overviews' do
-            expect(collection.push.first[:data][:assets])
-              .to eq(Overview.first.assets({}))
-          end
        end
      end
    end
--- a/spec/lib/user_info_spec.rb
+++ b/spec/lib/user_info_spec.rb
@@ -17,6 +17,8 @@ RSpec.describe UserInfo do

  describe '#ensure_current_user_id' do

+    let(:return_value) { 'Hello World' }
+
    it 'uses and keeps set User IDs' do
      test_id = 99
      described_class.current_user_id = test_id
@@ -37,5 +39,26 @@ RSpec.describe UserInfo do

      expect(described_class.current_user_id).to be nil
    end
+
+    it 'resets current_user_id in case of an exception' do
+      begin
+        described_class.ensure_current_user_id do
+          raise 'error'
+        end
+      rescue # rubocop:disable Lint/HandleExceptions
+      end
+
+      expect(described_class.current_user_id).to be nil
+    end
+
+    it 'passes return value of given block' do
+
+      received = described_class.ensure_current_user_id do
+        return_value
+      end
+
+      expect(received).to eq(return_value)
+    end
+
  end
 end
--- a/spec/models/channel/email_parser_spec.rb
+++ b/spec/models/channel/email_parser_spec.rb
@@ -194,7 +194,7 @@ RSpec.describe Channel::EmailParser, type: :model do
          RAW
        end

-        shared_context 'ticket reference in attachment' do
+        shared_context 'ticket reference in text/plain attachment' do
          let(:raw_mail) { <<~RAW.chomp }
            From: me@example.com
            Content-Type: multipart/mixed; boundary="Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2"
@@ -224,6 +224,96 @@ RSpec.describe Channel::EmailParser, type: :model do
          RAW
        end

+        shared_context 'ticket reference in text/html (as content) attachment' do
+          let(:raw_mail) { <<~RAW.chomp }
+            From: me@example.com
+            Content-Type: multipart/mixed; boundary="Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2"
+            Subject: no reference
+            Date: Sun, 30 Aug 2015 23:20:54 +0200
+            To: Martin Edenhofer <me@znuny.com>
+            Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
+            X-Mailer: Apple Mail (2.2104)
+
+
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2
+            Content-Transfer-Encoding: 7bit
+            Content-Type: text/plain;
+              charset=us-ascii
+
+            no reference
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2
+            Content-Disposition: attachment;
+              filename=test1.txt
+            Content-Type: text/html;
+              name="test.txt"
+            Content-Transfer-Encoding: 7bit
+
+            <div>Some Text #{ticket_ref}</div>
+
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2--
+          RAW
+        end
+
+        shared_context 'ticket reference in text/html (attribute) attachment' do
+          let(:raw_mail) { <<~RAW.chomp }
+            From: me@example.com
+            Content-Type: multipart/mixed; boundary="Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2"
+            Subject: no reference
+            Date: Sun, 30 Aug 2015 23:20:54 +0200
+            To: Martin Edenhofer <me@znuny.com>
+            Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
+            X-Mailer: Apple Mail (2.2104)
+
+
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2
+            Content-Transfer-Encoding: 7bit
+            Content-Type: text/plain;
+              charset=us-ascii
+
+            no reference
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2
+            Content-Disposition: attachment;
+              filename=test1.txt
+            Content-Type: text/html;
+              name="test.txt"
+            Content-Transfer-Encoding: 7bit
+
+            <div>Some Text <b data-something="#{ticket_ref}">some text</b></div>
+
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2--
+          RAW
+        end
+
+        shared_context 'ticket reference in image/jpg attachment' do
+          let(:raw_mail) { <<~RAW.chomp }
+            From: me@example.com
+            Content-Type: multipart/mixed; boundary="Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2"
+            Subject: no reference
+            Date: Sun, 30 Aug 2015 23:20:54 +0200
+            To: Martin Edenhofer <me@znuny.com>
+            Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
+            X-Mailer: Apple Mail (2.2104)
+
+
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2
+            Content-Transfer-Encoding: 7bit
+            Content-Type: text/plain;
+              charset=us-ascii
+
+            no reference
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2
+            Content-Disposition: attachment;
+              filename=test1.jpg
+            Content-Type: image/jpg;
+              name="test.jpg"
+            Content-Transfer-Encoding: 7bit
+
+            Some Text #{ticket_ref}
+
+            --Apple-Mail=_ED77AC8D-FB6F-40E5-8FBE-D41FF5E1BAF2--
+          RAW
+        end
+
        shared_context 'ticket reference in In-Reply-To header' do
          let(:raw_mail) { <<~RAW.chomp }
            From: me@example.com
@@ -328,8 +418,23 @@ RSpec.describe Channel::EmailParser, type: :model do
            include_examples 'creates a new ticket'
          end

-          context 'when attachment contains ticket reference' do
-            include_context 'ticket reference in attachment'
+          context 'when text/plain attachment contains ticket reference' do
+            include_context 'ticket reference in text/plain attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when text/html attachment (as content) contains ticket reference' do
+            include_context 'ticket reference in text/html (as content) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when text/html attachment (attribute) contains ticket reference' do
+            include_context 'ticket reference in text/html (attribute) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when image/jpg attachment contains ticket reference' do
+            include_context 'ticket reference in image/jpg attachment'
            include_examples 'creates a new ticket'
          end

@@ -448,8 +553,23 @@ RSpec.describe Channel::EmailParser, type: :model do
            end
          end

-          context 'when attachment contains ticket reference' do
-            include_context 'ticket reference in attachment'
+          context 'when text/plain attachment contains ticket reference' do
+            include_context 'ticket reference in text/plain attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when text/html attachment (as content) contains ticket reference' do
+            include_context 'ticket reference in text/html (as content) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when text/html attachment (attribute) contains ticket reference' do
+            include_context 'ticket reference in text/html (attribute) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when image/jpg attachment contains ticket reference' do
+            include_context 'ticket reference in image/jpg attachment'
            include_examples 'creates a new ticket'
          end

@@ -491,11 +611,26 @@ RSpec.describe Channel::EmailParser, type: :model do
            include_examples 'creates a new ticket'
          end

-          context 'when attachment contains ticket reference' do
-            include_context 'ticket reference in attachment'
+          context 'when text/plain attachment contains ticket reference' do
+            include_context 'ticket reference in text/plain attachment'
            include_examples 'adds message to ticket'
          end

+          context 'when text/html attachment (as content) contains ticket reference' do
+            include_context 'ticket reference in text/html (as content) attachment'
+            include_examples 'adds message to ticket'
+          end
+
+          context 'when text/html attachment (attribute) contains ticket reference' do
+            include_context 'ticket reference in text/html (attribute) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when image/jpg attachment contains ticket reference' do
+            include_context 'ticket reference in image/jpg attachment'
+            include_examples 'creates a new ticket'
+          end
+
          context 'when In-Reply-To header contains article message-id' do
            include_context 'ticket reference in In-Reply-To header'
            include_examples 'creates a new ticket'
@@ -534,8 +669,28 @@ RSpec.describe Channel::EmailParser, type: :model do
            include_examples 'creates a new ticket'
          end

-          context 'when attachment contains ticket reference' do
-            include_context 'ticket reference in attachment'
+          context 'when text/plain attachment contains ticket reference' do
+            include_context 'ticket reference in text/plain attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when text/html attachment (as content) contains ticket reference' do
+            include_context 'ticket reference in text/html (as content) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when text/html attachment (attribute) contains ticket reference' do
+            include_context 'ticket reference in text/html (attribute) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when image/jpg attachment contains ticket reference' do
+            include_context 'ticket reference in image/jpg attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when image/jpg attachment contains ticket reference' do
+            include_context 'ticket reference in image/jpg attachment'
            include_examples 'creates a new ticket'
          end

@@ -592,11 +747,26 @@ RSpec.describe Channel::EmailParser, type: :model do
            include_examples 'adds message to ticket'
          end

-          context 'when attachment contains ticket reference' do
-            include_context 'ticket reference in attachment'
+          context 'when text/plain attachment contains ticket reference' do
+            include_context 'ticket reference in text/plain attachment'
            include_examples 'adds message to ticket'
          end

+          context 'when text/html attachment (as content) contains ticket reference' do
+            include_context 'ticket reference in text/html (as content) attachment'
+            include_examples 'adds message to ticket'
+          end
+
+          context 'when text/html attachment (attribute) contains ticket reference' do
+            include_context 'ticket reference in text/html (attribute) attachment'
+            include_examples 'creates a new ticket'
+          end
+
+          context 'when image/jpg attachment contains ticket reference' do
+            include_context 'ticket reference in image/jpg attachment'
+            include_examples 'creates a new ticket'
+          end
+
          context 'when In-Reply-To header contains article message-id' do
            include_context 'ticket reference in In-Reply-To header'
            include_examples 'adds message to ticket'
@@ -773,6 +943,18 @@ RSpec.describe Channel::EmailParser, type: :model do
            .not_to raise_error
        end
      end
+
+      context 'when attachment for follow up check contains invalid charsets (#2808)' do
+        let(:mail_file) { Rails.root.join('test', 'data', 'mail', 'mail085.box') }
+
+        before { Setting.set('postmaster_follow_up_search_in', %w[attachment body]) }
+
+        it 'does not raise Encoding::CompatibilityError:' do
+          expect { described_class.new.process({}, raw_mail) }
+            .not_to raise_error
+        end
+      end
+
    end

    describe 'attachment handling' do
--- a/spec/models/channel/filter/follow_up_possible_check_spec.rb
+++ b/spec/models/channel/filter/follow_up_possible_check_spec.rb
@@ -0,0 +1,37 @@
+require 'rails_helper'
+
+RSpec.describe Channel::Filter::FollowUpPossibleCheck, type: :channel_filter do
+
+  context 'new_ticket follow_up_possible for group' do
+
+    let(:group) { create(:group, follow_up_possible: 'new_ticket') }
+
+    context 'ticket closed' do
+      let(:ticket) { create(:ticket, group: group, state: Ticket::State.find_by(name: 'closed')) }
+
+      it 'prevents follow up' do
+        mail = {
+          'x-zammad-ticket-id': ticket.id
+        }
+
+        filter(mail)
+
+        expect(mail[:'x-zammad-ticket-id']).not_to eq(ticket.id)
+      end
+    end
+
+    context 'ticket open' do
+      let(:ticket) { create(:ticket, group: group, state: Ticket::State.find_by(name: 'new')) }
+
+      it 'allows follow up' do
+        mail = {
+          'x-zammad-ticket-id': ticket.id
+        }
+
+        filter(mail)
+
+        expect(mail[:'x-zammad-ticket-id']).to eq(ticket.id)
+      end
+    end
+  end
+end
--- a/spec/models/object_manager/attribute_spec.rb
+++ b/spec/models/object_manager/attribute_spec.rb
@@ -57,10 +57,20 @@ RSpec.describe ObjectManager::Attribute, type: :model do
      end.to raise_error 'attribute is a reserved word, please choose a different one'
    end

-    it 'rejects Zammad reserved word "table"' do
-      expect do
-        ObjectManager::Attribute.add attributes_for :object_manager_attribute_text, name: 'table'
-      end.to raise_error 'table is a reserved word, please choose a different one'
+    %w[destroy true false integer select drop create alter index table varchar blob date datetime timestamp url icon initials avatar permission validate subscribe unsubscribe translate search _type _doc _id id].each do |reserved_word|
+      it "rejects Zammad reserved word '#{reserved_word}'" do
+        expect do
+          ObjectManager::Attribute.add attributes_for :object_manager_attribute_text, name: reserved_word
+        end.to raise_error "#{reserved_word} is a reserved word, please choose a different one"
+      end
+    end
+
+    %w[someting_id something_ids].each do |reserved_word|
+      it "rejects word '#{reserved_word}' which is used for database references" do
+        expect do
+          ObjectManager::Attribute.add attributes_for :object_manager_attribute_text, name: reserved_word
+        end.to raise_error "Name can't get used, *_id and *_ids are not allowed"
+      end
    end

    it 'rejects duplicate attribute name of conflicting types' do
--- a/spec/models/ticket/subject_spec.rb
+++ b/spec/models/ticket/subject_spec.rb
@@ -0,0 +1,51 @@
+require 'rails_helper'
+
+RSpec.describe Ticket::Subject do
+  let(:ticket) { create(:ticket) }
+
+  describe '.subject_build' do
+    it 'build subject based on new title' do
+      expect(ticket.subject_build('other title')).to eq("other title [Ticket##{ticket.number}]")
+    end
+    it 'build subject based on new title with ticket_hook_position left' do
+      Setting.set('ticket_hook_position', 'left')
+      expect(ticket.subject_build('other title')).to eq("[Ticket##{ticket.number}] other title")
+    end
+
+    it 'build subject based on new title without ticket_hook_position' do
+      Setting.set('ticket_hook_position', '')
+      expect(ticket.subject_build('other title')).to eq('other title')
+    end
+    it 'build subject based with forward argument' do
+      expect(ticket.subject_build('other title', 'forward')).to eq("FWD: other title [Ticket##{ticket.number}]")
+    end
+    it 'build subject based with reply argument' do
+      expect(ticket.subject_build('other title', 'reply')).to eq("RE: other title [Ticket##{ticket.number}]")
+    end
+  end
+
+  describe '.subject_clean' do
+    it 'cleanup subject with undefined string' do
+      expect(ticket.subject_clean(nil)).to eq('')
+    end
+    it 'cleanup subject with empty string' do
+      expect(ticket.subject_clean('')).to eq('')
+    end
+    it 'cleanup subject with long string which need to be truncated by [...]' do
+      expect(ticket.subject_clean('123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890')).to eq('12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890[...]')
+    end
+    it 'cleanup subject with regular ticket#' do
+      expect(ticket.subject_clean("something [Ticket##{ticket.number}]")).to eq('something')
+    end
+    it 'cleanup subject with regular ticket# multiple time' do
+      expect(ticket.subject_clean("[Ticket##{ticket.number}] [Ticket##{ticket.number}] something [Ticket##{ticket.number}]")).to eq('something')
+    end
+    it 'cleanup subject with foreign ticket#' do
+      expect(ticket.subject_clean('something [Ticket#123456]')).to eq('something [Ticket#123456]')
+    end
+    it 'cleanup subject with some reply signs' do
+      expect(ticket.subject_clean('RE: RE: Re[5]: something [Ticket#123456]')).to eq('something [Ticket#123456]')
+    end
+  end
+
+end
--- a/spec/system/js/q_unit_spec.rb
+++ b/spec/system/js/q_unit_spec.rb
@@ -47,6 +47,10 @@ RSpec.describe 'QUnit', type: :system, authenticated: false, set_up: true, webso
      q_unit_tests('model_ui')
    end

+    it 'Model Ticket' do
+      q_unit_tests('model_ticket')
+    end
+
    it 'Ticket selector' do
      q_unit_tests('ticket_selector')
    end
--- a/test/browser_test_helper.rb
+++ b/test/browser_test_helper.rb
@@ -2663,7 +2663,14 @@ wait untill text in selector disabppears
           end

    # switch to overview
-    instance.find_elements(css: ".content.active .sidebar a[href=\"#{link}\"]")[0].click
+    element = nil
+    6.times do
+      element = instance.find_elements(css: ".content.active .sidebar a[href=\"#{link}\"]")[0]
+      break if element
+
+      sleep 1
+    end
+    element.click

    # hide larger overview selection list again
    sleep 0.5
@@ -2698,14 +2705,25 @@ wait untill text in selector disabppears

    overview_open(params)

+    element = nil
    if params[:title]
-      element = instance.find_element(css: '.content.active').find_element(partial_link_text: params[:title])
+      6.times do
+        element = instance.find_element(css: '.content.active').find_element(partial_link_text: params[:title])
+        break if element
+
+        sleep 1
+      end
      if !element
        screenshot(browser: instance, comment: 'ticket_open_by_overview_no_ticket_failed')
        raise "unable to find ticket #{params[:title]} in overview #{params[:link]}!"
      end
    else
-      element = instance.find_elements(partial_link_text: params[:number])[0]
+      6.times do
+        element = instance.find_elements(partial_link_text: params[:number])[0]
+        break if element
+
+        sleep 1
+      end
      if !element
        screenshot(browser: instance, comment: 'ticket_open_by_overview_no_ticket_failed')
        raise "unable to find ticket #{params[:number]} in overview #{params[:link]}!"
--- a/test/data/mail/mail085.box
+++ b/test/data/mail/mail085.box
@@ -0,0 +1,80 @@
+Return-Path: <bob.smith@example.com>
+From: Bob Smith <bob.smith@example.com>
+Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
+Content-Type: multipart/alternative;
+	boundary="Apple-Mail=_E59F63A4-B0CE-4985-BD59-45DAA0F6596C"
+X-Priority: 3
+Subject: =?utf-8?Q?Re=3A_buy_-_Wir_freuen_uns_=C3=BCber_Deine_kurze_Be?=
+ =?utf-8?Q?wertung!_=231XXXX9?=
+Date: Wed, 13 Nov 2019 12:11:56 +0100
+References: <01cf9cbc1a5d1b6f2bd07b26a2bbb896@www.example.com>
+To: "Support GmbH" <kaufs@example.com>
+In-Reply-To: <01cf9cbc1a5d1b6f2bd07b26a2bbb896@www.example.com>
+Message-Id: <4A8DD423-FC24-4CC5-8996-4E27A57BA774@example.com>
+X-Mailer: Apple Mail (2.3445.104.11)
+Envelope-To: <kaufs@example.com>
+X-Spam-Flag: NO
+
+
+--Apple-Mail=_E59F63A4-B0CE-4985-BD59-45DAA0F6596C
+Content-Transfer-Encoding: quoted-printable
+Content-Type: text/plain;
+	charset=utf-8
+
+Liebes Team,
+
+Mit freundlichen Gr=C3=BC=C3=9Fen
+_____________________________________
+
+Office Management                                    =20
+
+
+--Apple-Mail=_E59F63A4-B0CE-4985-BD59-45DAA0F6596C
+Content-Type: multipart/mixed;
+	boundary="Apple-Mail=_E53D4560-0E2F-48BE-B7E3-427332B2933C"
+
+
+--Apple-Mail=_E53D4560-0E2F-48BE-B7E3-427332B2933C
+Content-Transfer-Encoding: quoted-printable
+Content-Type: text/html;
+	charset=utf-8
+
+123
+
+--Apple-Mail=_E53D4560-0E2F-48BE-B7E3-427332B2933C
+Content-Transfer-Encoding: quoted-printable
+Content-Type: text/html;
+	charset=utf-8
+
+<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; =
+charset=2Dutf-8"></head><body>
+<div>_____________________________________</div><div><br =
+class=3D"">11111 Someware<br class=3D""><br class=3D"">Telefon: +49.(0) =
+30.1234<br class=3D"">Fax: +49.(0) 30.1234<br class=3D"">
+HRB 12345 B<br class=3D""><br class=3D"">Gesch=C3=A4ftsf=C3=BChrung: =
+Bob Smith</div></div></div>
+</div>
+<div><br class=3D""><blockquote type=3D"cite" class=3D"">
+<div class=3D"">
+    <div class=3D"">
+    <table border=3D"0" cellspacing=3D"0" cellpadding=3D"0" width=3D"600" =
+class=3D"">
+        <tbody class=3D""><tr class=3D"">
+        <tr class=3D"">
+            <td style=3D"font-family: Arial, Helvetica, =
+sans-serif;font-size: 12px;line-height:18px;color:#333" class=3D""><div =
+Antworte  einfach auf diese E-Mail und schildere kurz, was passiert ist, =
+damit wir uns verbessern k=C3=B6nnen!<br class=3D"">
+<br class=3D"">
+Dann w=C3=BCrden wir uns unheimlich freuen, wenn Du Dir <strong =
+class=3D"">
+            </td>
+        </tr>
+    </tbody></table>
+    </div>
+
+</div></blockquote></div><br class=3D""></div></body></html>=
+
+--Apple-Mail=_E53D4560-0E2F-48BE-B7E3-427332B2933C--
+
+--Apple-Mail=_E59F63A4-B0CE-4985-BD59-45DAA0F6596C--
--- a/test/integration/email_deliver_test.rb
+++ b/test/integration/email_deliver_test.rb
@@ -11,12 +11,16 @@ class EmailDeliverTest < ActiveSupport::TestCase
      raise "Need MAIL_SERVER_ACCOUNT as ENV variable like export MAIL_SERVER_ACCOUNT='user:somepass'"
    end

+    if ENV['MAIL_SERVER_EMAIL'].blank?
+      raise "Need MAIL_SERVER_EMAIL as ENV variable like export MAIL_SERVER_EMAIL='someunitest@example.com'"
+    end
+
    server_login = ENV['MAIL_SERVER_ACCOUNT'].split(':')[0]
    server_password = ENV['MAIL_SERVER_ACCOUNT'].split(':')[1]

    email_address = EmailAddress.create!(
      realname:      'me Helpdesk',
-      email:         "me#{rand(999_999_999)}@example.com",
+      email:         "some-zammad-#{ENV['MAIL_SERVER_EMAIL']}",
      updated_by_id: 1,
      created_by_id: 1,
    )
--- a/test/unit/email_process_test.rb
+++ b/test/unit/email_process_test.rb
@@ -3180,6 +3180,31 @@ Content-Type: text/html; charset=us-ascii; format=flowed
  end

  test 'process trusted' do
+    groups = Group.all
+    roles  = Role.where(name: 'Agent')
+    agent1 = User.create!(
+      login:         'agent1',
+      firstname:     'Firstname',
+      lastname:      'agent1',
+      email:         'agent1@example.com',
+      active:        true,
+      roles:         roles,
+      groups:        groups,
+      updated_by_id: 1,
+      created_by_id: 1,
+    )
+    roles  = Role.where(name: 'Customer')
+    customer1 = User.create!(
+      login:         'customer1',
+      firstname:     'Firstname',
+      lastname:      'customer1',
+      email:         'customer1@example.com',
+      active:        true,
+      roles:         roles,
+      updated_by_id: 1,
+      created_by_id: 1,
+    )
+
    files = [
      {
        data: 'From: me@example.com
@@ -3199,6 +3224,7 @@ To: customer@example.com
 Subject: some subject
 X-Zammad-Ticket-Followup-State: closed
 X-Zammad-Ticket-priority: 3 high
+X-Zammad-Ticket-owner: agent1@example.com 
 X-Zammad-Article-sender: System
 x-Zammad-Article-type: phone
 x-Zammad-Article-Internal: true
@@ -3213,6 +3239,7 @@ Some Text',
            state: 'new',
            priority: '3 high',
            title: 'some subject',
+            owner: agent1,
          },
          1 => {
            sender: 'System',
@@ -3227,6 +3254,7 @@ To: customer@example.com
 Subject: some subject
 X-Zammad-Ticket-Followup-State: closed
 X-Zammad-Ticket-priority_id: 777777
+X-Zammad-Ticket-owner: not_existing@example.com 
 X-Zammad-Article-sender_id: 999999
 x-Zammad-Article-type: phone
 x-Zammad-Article-Internal: true
@@ -3241,6 +3269,7 @@ Some Text',
            state: 'new',
            priority: '2 normal',
            title: 'some subject',
+            owner: User.find(1),
          },
          1 => {
            sender: 'Customer',
@@ -3249,6 +3278,81 @@ Some Text',
          },
        },
      },
+      {
+        data: 'From: me@example.com
+To: customer@example.com
+Subject: some subject / with customer as agent - customer can not be owner
+X-Zammad-Ticket-owner: customer1@example.com 
+
+Some Text',
+        channel: {
+          trusted: true,
+        },
+        success: true,
+        result: {
+          0 => {
+            state: 'new',
+            priority: '2 normal',
+            title: 'some subject / with customer as agent - customer can not be owner',
+            owner: User.find(1),
+          },
+          1 => {
+            sender: 'Customer',
+            type: 'email',
+            internal: false,
+          },
+        },
+      },
+      {
+        data: 'From: me@example.com
+To: customer@example.com
+Subject: some subject / with agent login
+X-Zammad-Ticket-owner: agent1
+
+Some Text',
+        channel: {
+          trusted: true,
+        },
+        success: true,
+        result: {
+          0 => {
+            state: 'new',
+            priority: '2 normal',
+            title: 'some subject / with agent login',
+            owner: agent1,
+          },
+          1 => {
+            sender: 'Customer',
+            type: 'email',
+            internal: false,
+          },
+        },
+      },
+      {
+        data: 'From: me@example.com
+To: customer@example.com
+Subject: some subject / with agent email
+X-Zammad-Ticket-owner: agent1@example.com
+
+Some Text',
+        channel: {
+          trusted: true,
+        },
+        success: true,
+        result: {
+          0 => {
+            state: 'new',
+            priority: '2 normal',
+            title: 'some subject / with agent email',
+            owner: agent1,
+          },
+          1 => {
+            sender: 'Customer',
+            type: 'email',
+            internal: false,
+          },
+        },
+      },
    ]
    assert_process(files)
  end
Author	SHA1	Message	Date
Martin Edenhofer	816c67ab95	Updated changes to 3.1.1.	2019-11-29 15:17:25 +01:00
Thorsten Eckel	3f54835b4e	Fixed issue #2808 - Unprocessable mail if follow up search is enabled for attachments with contains invalid charsets (postmaster_follow_up_search_in with attachment).	2019-11-18 18:53:39 +01:00
Martin Edenhofer	99351f6529	Fixed issue #2796 - Image/attachment preview is to small, need to download image and open it locally.	2019-11-18 18:50:40 +01:00
Martin Edenhofer	6fba2c9cbf	Fixed bug: OTRS import - User queue/group permissions are not transfered depending on export data type.	2019-11-18 17:04:32 +01:00
Martin Edenhofer	4c8a5ba42a	Maintenance: Updated nokogiri gem - CVE-2019-13117	2019-11-18 16:32:41 +01:00
Martin Edenhofer	5878cb4bfe	Performance: `Sessions::Backend::TicketOverivewList` check whether data has to be sent to the client was wrong. The correct check now compares the list of tickets instead of comparing the whole hash structure including assets.	2019-11-07 13:30:41 +01:00
Mantas	b7bfdcd47a	Fixed bug: ExcelSheet timezone conversion doesn't take DST into account.	2019-10-29 19:36:13 +01:00
Thorsten Eckel	75b0721cb5	Maintenance: Enabled support for Debian 10 buster package building on packager.io.	2019-10-29 17:36:25 +01:00
Thorsten Eckel	604a6b6027	Bumped loofah to 2.3.1 (CVE-2019-15587).	2019-10-29 17:36:11 +01:00
Thorsten Eckel	c1160c5ee8	Fixes #2768 - Unable to process email where group has option follow_up_possible: 'new_ticket' and arriving email has no subject header.	2019-10-08 16:40:28 +02:00
Olivier Sallou	1a1a4dd01a	Fix typos in Unprocessable Entity error message 🎉🎉🎉 Big thanks to Olivier Sallou (@osallou) for this contribution!	2019-10-08 17:50:17 +08:00
Martin Edenhofer	198d7469df	Fixes #2699 - Excel export of report or time accounting drops additional fields in row if integer field contain `null/nil/undefined`.	2019-10-07 16:47:35 +02:00
Martin Edenhofer	d68650e2c7	Updated rubyzip / CVE-2019-16892	2019-09-30 15:54:39 +02:00
Thorsten Eckel	b232e9fb4c	Fixed issue #2742 - Removing a record drops whole index on Elasticsearch 6 and later.	2019-09-27 00:11:47 +02:00
Thorsten Eckel	6fa1378f3c	Fixed issue #2750 - Can't change Ticket attributes of "Shared Organization" Ticket.	2019-09-20 10:12:37 +02:00
Thorsten Eckel	19ffb6096f	Fixed mail delivery postmaster to sender test.	2019-09-18 16:17:46 +02:00
Thorsten Eckel	4f556a4902	fix typos in html tags This will satsify issue https://github.com/zammad/zammad/issues/2747 found by abeluck. Earlier the two patched files did not correctly close divs, which caused broken Mails to be generated.	2019-09-18 14:41:01 +02:00
Martin Edenhofer	7289266cd3	Fixed issue #2728 - Put _type, _doc and _id to reserved words for object managaer becasue of elasticssearch internal use. .	2019-09-03 01:18:45 +02:00
Martin Edenhofer	396a40e206	Follow up for issue #2720 - makes Model.search_index_reload more robust - Sometimes elasticsearch takes longer as expected and will raise timeout in Zammad.	2019-09-03 01:14:19 +02:00
Martin Edenhofer	8479deb08b	Fized issue #2723 - Unable to process mail with enabled follow up detection in content of attachments.	2019-08-30 16:02:14 +02:00
Martin Edenhofer	4a5edb772d	Fixed issue #2722 - mail processing impossible: invalid byte sequence in UTF-8.	2019-08-29 19:00:21 +02:00
Martin Edenhofer	7f1a9f75e6	Martin Edenhofer's avatar Fixed issue #2719 - Unable to set owner via `X-Zammad-Ticket-owner` header.	2019-08-29 10:00:47 +02:00
Martin Edenhofer	109d74bb84	Fixed issue #2720 - Sometimes es takes longer as expected and will raise timeout in Zammad.	2019-08-28 12:46:33 +02:00
Thorsten Eckel	827b1a4130	Maintenance: Improved HTML sanitization timeout handling.	2019-08-16 16:21:04 +02:00
Thorsten Eckel	2de61bd4c5	Bumped nokogiri to 1.10.4 (CVE-2019-5477).	2019-08-14 08:37:02 +02:00
Martin Edenhofer	fe6464232c	Fixed issue #2697 - Time zone incorrectly calculated in ticket list.	2019-08-09 13:43:04 +02:00
Thorsten Eckel	b2f434c570	Fixed bug #2675 : Follow up to `9503ff20ce` - Linking auth provider account from profile fails.	2019-07-29 14:39:14 +02:00
Thorsten Eckel	fb3e2d49c4	Fixed issue #2663 : DB migration blocks update when records with 'current_user.id' in selector of Job or Trigger exist.	2019-07-12 16:19:40 +02:00
Thorsten Eckel	f7166ee40a	Refactoring UserInfo: - Pass return value of given block back to caller. - Ensure that exceptions don't cause the temporary UserInfo.current_user_id to leak.	2019-07-12 16:19:40 +02:00
MrGeneration	7db565d7ea	Unset RAILS_LOG_TO_STDOUT by default This will solve issue #2660 and ensure that we're not changing the logging by default. Set RAILS_LOG_TO_STDOUT=true to enable logging to STDOUT.	2019-07-12 12:14:51 +02:00
Ryan Lue	669d0bebe3	Support NullDB adapter (used in Docker packaging) in DBPreflightCheck initializer	2019-07-12 12:07:00 +02:00
Denny Bresch	aa61d8ddd0	See more: make last row of text selectable fixes #2654	2019-07-11 19:20:15 +02:00
Thorsten Eckel	7a10750f2f	Applied workaround for issue #2656 : Disable new Rails database connection reaping functionality until the Thread-Zombie issue is resolved: https://github.com/rails/rails/issues/33600 .	2019-07-10 14:04:59 +02:00
Martin Edenhofer	2a871aebd5	Updated changes to 3.1.0.	2019-07-10 01:18:32 +02:00