Updated.

Conflicts:
	config/routes/form.rb

.gitlab-ci.yml

@@ -330,7 +330,7 @@ test:integration:otrs_5_mysql:
     - mysql
   script:
     - export RAILS_ENV=test
-    - export IMPORT_OTRS_ENDPOINT="http://vz599.demo.znuny.com/otrs/public.pl?Action=ZammadMigrator"
+    - export IMPORT_OTRS_ENDPOINT="http://vz1109.demo.znuny.com/otrs/public.pl?Action=ZammadMigrator"
     - rake db:create
     - rake db:migrate
     - ruby -I test/ test/integration/otrs_import_test.rb
@@ -343,7 +343,7 @@ test:integration:otrs_5_postgresql:
     - postgresql
   script:
     - export RAILS_ENV=test
-    - export IMPORT_OTRS_ENDPOINT="http://vz599.demo.znuny.com/otrs/public.pl?Action=ZammadMigrator"
+    - export IMPORT_OTRS_ENDPOINT="http://vz1109.demo.znuny.com/otrs/public.pl?Action=ZammadMigrator"
     - rake db:create
     - rake db:migrate
     - ruby -I test/ test/integration/otrs_import_test.rb
@@ -409,6 +409,7 @@ browser:build:
     - rake assets:precompile
     - rake db:drop
   artifacts:
+    expire_in: 1 week
     paths:
     - public/assets/.sprockets-manifest*
     - public/assets/application-*

.travis.yml

@@ -55,3 +55,4 @@ script:
   - ruby -I test/ test/integration/user_device_controller_test.rb
   - ruby -I test/ test/integration/sipgate_controller_test.rb
   - rake db:drop
+after_success: contrib/travis-ci.org/trigger-docker-compose-build.sh

CHANGELOG.md

@@ -1,16 +1,45 @@
 # Change Log
 
-## [1.5.0](https://github.com/zammad/zammad/tree/1.5.0) (2017-xx-xx)
-[Full Changelog](https://github.com/zammad/zammad/compare/1.4.0...1.5.0)
-
-**Implemented enhancements:**
-
-
-
+## [1.5.1](https://github.com/zammad/zammad/tree/1.5.1) (2017-09-11)
+[Full Changelog](https://github.com/zammad/zammad/compare/1.5.0...1.5.1)
 
 **Fixed bugs:**
 
-
-
+- Broken email with huge pdf inline \(lead to huge article body with bigger then 7MB\) is blocking rendering of web app [\#1390](https://github.com/zammad/zammad/issues/1390) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Macro/Bulk action: Cannot transfer more than one ticket to agent [\#1107](https://github.com/zammad/zammad/issues/1107) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Duplicates in Caller Log \(Sipgate\) [\#1253](https://github.com/zammad/zammad/issues/1253) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Time Accounting not showing all tickets, time is missing completly of specific ticket [\#1315](https://github.com/zammad/zammad/issues/1315) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- 'Access-Control-Allow-Origin' error when submitting web form. [\#1318](https://github.com/zammad/zammad/issues/1318) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Ticket Hook Position between RE: and the ticket subject [\#1369](https://github.com/zammad/zammad/issues/1369) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- OTRS Import fails with "incompatible character encodings: ASCII-8BIT and UTF-8 \(Encoding::CompatibilityError\)" [\#1349](https://github.com/zammad/zammad/issues/1349) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- DynamicFields imported from OTRS are mandatory, hidden and stop input forms from submitting [\#1175](https://github.com/zammad/zammad/issues/1175) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- handling of incoming mails seems strange / customer is not set correctly [\#1351](https://github.com/zammad/zammad/issues/1351) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- LDAP group retrieval fails for objectClass 'organization' [\#1347](https://github.com/zammad/zammad/issues/1347) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Zendesk import fails for deleted tickets [\#1161](https://github.com/zammad/zammad/issues/1161) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Missing replacement of Config and Current User objects in text modules and signatures [\#1290](https://github.com/zammad/zammad/issues/1290) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Chat widget error "chat notice || Translation 'es' needed!" [\#1267](https://github.com/zammad/zammad/issues/1267) [[enhancement](https://github.com/zammad/zammad/labels/enhancement)]
+- Tel Protocol Link for Telephone and Mobile [\#991](https://github.com/zammad/zammad/issues/991) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Don't use Proxy on localhost Address. [\#1292](https://github.com/zammad/zammad/issues/1292) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Race condition if agents merge ticket at same time but in different directions. [\#1216](https://github.com/zammad/zammad/issues/1216) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Block escape key when creating / editing text modules [\#1291](https://github.com/zammad/zammad/issues/1291) [[bug](https://github.com/zammad/zammad/labels/bug)] [[UX/UI](https://github.com/zammad/zammad/labels/UX/UI)]
+- Problem with cyrillic letters in name of overviews. [\#1151](https://github.com/zammad/zammad/issues/1151) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Backslash issue [\#1230](https://github.com/zammad/zammad/issues/1230) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- OTRS import fails for attributes containing null bytes strings on PostgreSQL [\#1200](https://github.com/zammad/zammad/issues/1200) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Ticket create state after OTRS import is wrong [\#1184](https://github.com/zammad/zammad/issues/1184) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Lost LDAP users are still valid after sync [\#1211](https://github.com/zammad/zammad/issues/1211) [[enhancement](https://github.com/zammad/zammad/labels/enhancement)]
+- LDAP - Role assignment based on groups does not work [\#1179](https://github.com/zammad/zammad/issues/1179) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Generation of weekly report fails with internal server error [\#1176](https://github.com/zammad/zammad/issues/1176) [[bug](https://github.com/zammad/zammad/labels/bug)] [[reporting](https://github.com/zammad/zammad/labels/reporting)]
+- Zendesk import: uninitialized constant Import::Zendesk::ObjectAttribute::Basic\_priority [\#1153](https://github.com/zammad/zammad/issues/1153) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Configuration of LDAP with disabled anonymous bind fails [\#1114](https://github.com/zammad/zammad/issues/1114) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Group and User filter detection for freeIPA LDAP fails [\#1155](https://github.com/zammad/zammad/issues/1155) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Zendesk import: Error while importing ticket field with dash in name [\#1095](https://github.com/zammad/zammad/issues/1095) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- LDAP: mapping LDAP groups to multiple Zammad roles [\#1097](https://github.com/zammad/zammad/issues/1097) [[enhancement](https://github.com/zammad/zammad/labels/enhancement)]
+- Email loop if default trigger for follow up is active and customer email address is invalid [\#1131](https://github.com/zammad/zammad/issues/1131) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Update of object attribute leads to CSRF token verification failure [\#1034](https://github.com/zammad/zammad/issues/1034) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Not showing TO: / CC: fields on IE11 when answering a ticket [\#764](https://github.com/zammad/zammad/issues/764) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Internet Explorer 11 and lower - no paste from Clipboard possible [\#990](https://github.com/zammad/zammad/issues/990) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Tag Bug - tag is deleted if renamed without changes via admin interface [\#1086](https://github.com/zammad/zammad/issues/1086) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- Unprocessable Email \(without email address in from header\) [\#811](https://github.com/zammad/zammad/issues/811) [[bug](https://github.com/zammad/zammad/labels/bug)]
+- LDAP Integration eDirectoy doesn't find user entries/attributes [\#1088](https://github.com/zammad/zammad/issues/1088) [[bug](https://github.com/zammad/zammad/labels/bug)]
 
 \* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*

Gemfile.lock

@@ -61,7 +61,7 @@ GEM
     childprocess (0.5.9)
       ffi (~> 1.0, >= 1.0.11)
     clavius (1.0.2)
-    clearbit (0.2.5)
+    clearbit (0.2.7)
       nestful (~> 1.1.0)
     coderay (1.1.1)
     coffee-rails (4.2.1)
@@ -156,7 +156,7 @@ GEM
     guard-symlink (0.1.0)
       guard-compat (~> 1.1)
     hashdiff (0.3.2)
-    hashie (3.4.4)
+    hashie (3.5.6)
     htmlentities (4.3.4)
     http (1.0.4)
       addressable (~> 2.3)
@@ -171,7 +171,7 @@ GEM
     icalendar (2.4.1)
     inflection (1.0.0)
     json (1.8.6)
-    jwt (1.5.4)
+    jwt (1.5.6)
     kgio (2.11.0)
     koala (2.4.0)
       addressable
@@ -193,10 +193,10 @@ GEM
     mime-types (3.1)
       mime-types-data (~> 3.2015)
     mime-types-data (3.2016.0521)
-    mini_portile2 (2.1.0)
+    mini_portile2 (2.2.0)
     minitest (5.10.1)
     multi_json (1.12.1)
-    multi_xml (0.5.5)
+    multi_xml (0.6.0)
     multipart-post (2.0.0)
     mysql2 (0.4.4)
     naught (1.1.0)
@@ -204,33 +204,33 @@ GEM
     nestful (1.1.1)
     net-ldap (0.15.0)
     netrc (0.11.0)
-    nokogiri (1.7.1)
-      mini_portile2 (~> 2.1.0)
+    nokogiri (1.8.0)
+      mini_portile2 (~> 2.2.0)
     notiffany (0.1.1)
       nenv (~> 0.1)
       shellany (~> 0.0)
     oauth (0.5.1)
-    oauth2 (1.2.0)
-      faraday (>= 0.8, < 0.10)
+    oauth2 (1.4.0)
+      faraday (>= 0.8, < 0.13)
       jwt (~> 1.0)
       multi_json (~> 1.3)
       multi_xml (~> 0.5)
       rack (>= 1.2, < 3)
     octokit (4.4.1)
       sawyer (~> 0.7.0, >= 0.5.3)
-    omniauth (1.3.1)
-      hashie (>= 1.2, < 4)
-      rack (>= 1.0, < 3)
+    omniauth (1.6.1)
+      hashie (>= 3.4.6, < 3.6.0)
+      rack (>= 1.6.2, < 3)
     omniauth-facebook (4.0.0)
       omniauth-oauth2 (~> 1.2)
-    omniauth-github (1.1.2)
-      omniauth (~> 1.0)
-      omniauth-oauth2 (~> 1.1)
+    omniauth-github (1.3.0)
+      omniauth (~> 1.5)
+      omniauth-oauth2 (>= 1.4.0, < 2.0)
     omniauth-gitlab (1.0.2)
       omniauth (~> 1.0)
       omniauth-oauth2 (~> 1.0)
-    omniauth-google-oauth2 (0.4.1)
-      jwt (~> 1.5.2)
+    omniauth-google-oauth2 (0.5.0)
+      jwt (~> 1.5)
       multi_json (~> 1.3)
       omniauth (>= 1.1.1)
       omniauth-oauth2 (>= 1.3.1)
@@ -339,7 +339,6 @@ GEM
     sawyer (0.7.0)
       addressable (>= 2.3.5, < 2.5)
       faraday (~> 0.8, < 0.10)
-    scrub_rb (1.0.1)
     selenium-webdriver (2.53.4)
       childprocess (~> 0.5)
       rubyzip (~> 1.0)
@@ -408,13 +407,12 @@ GEM
       hashdiff
     websocket (1.2.3)
     writeexcel (1.0.5)
-    zendesk_api (1.14.0)
+    zendesk_api (1.14.4)
       faraday (~> 0.9)
-      hashie (>= 1.2, < 4.0, != 3.3.0)
+      hashie (>= 3.5.2, < 4.0.0)
       inflection
       mime-types
       multipart-post (~> 2.0)
-      scrub_rb (~> 1.0.1)
 
 PLATFORMS
   ruby

VERSION

@@ -1 +1 @@
-1.4.x
+1.5.x

app/assets/javascripts/app/controllers/_application_controller_table.coffee

@@ -225,65 +225,46 @@ class App.ControllerTable extends App.Controller
         events:
           click: @deleteRow
 
-    if @orderDirection && @orderBy
-      for header in @headers
-        if header.name is @orderBy
-          @objects = _.sortBy(
-            @objects
-            (item) ->
-              # if we need to sort translated col.
-              if header.translate
-                App.i18n.translateInline(item[header.name])
-              # if we need to sort a relation
-              if header.relation
-                if item[header.name]
-                  App[header.relation].findNative(item[header.name]).displayName()
-                else
-                  ''
-              else
-                item[header.name]
-          )
-          if @orderDirection is 'DESC'
-            header.sortOrderIcon = ['arrow-down', 'table-sort-arrow']
-            @objects = @objects.reverse()
-          else
-            header.sortOrderIcon = ['arrow-up', 'table-sort-arrow']
-        else
-          header.sortOrderIcon = undefined
+    if @orderDirection && @orderBy && !@groupBy
+      @objects = @sortList(@objects)
+
+    # group by
+    if @groupBy
+
+      # get groups
+      groupObjects = {}
+      for object in @objects
+        group = object[@groupBy]
+        if !group
+          withId = "#{@groupBy}_id"
+
+          if object[withId] && attributes[withId] && attributes[withId].relation
+            if App[attributes[withId].relation].exists(object[withId])
+              item = App[attributes[withId].relation].findNative(object[withId])
+              if item && item.displayName
+                group = item.displayName().toLowerCase()
+        if _.isEmpty(group)
+          group = ''
+        groupObjects[group] ||= []
+        groupObjects[group].push object
+
+      groupsSorted = []
+      for key of groupObjects
+        groupsSorted.push key
+      groupsSorted = groupsSorted.sort()
+
+      # get new order
+      @objects = []
+      for group in groupsSorted
+        localObjects = @sortList(groupObjects[group])
+        @objects = @objects.concat localObjects
+        groupObjects[group] = [] # release old array
 
     # execute header callback
     if @callbackHeader
       for callback in @callbackHeader
         @headers = callback(@headers)
 
-    # group by
-    if @groupBy
-
-      # get new order
-      groupObjects = _.groupBy(
-        @objects
-        (item) =>
-          return '' if !item[@groupBy]
-          return item[@groupBy].displayName() if item[@groupBy].displayName
-          item[@groupBy]
-      )
-      groupOrder = []
-      for group, value of groupObjects
-        groupOrder.push group
-
-      # sort new groups
-      groupOrder = _.sortBy(
-        groupOrder
-        (item) ->
-          item
-      )
-
-      # create new data array
-      @objects = []
-      for group in groupOrder
-        @objects = @objects.concat groupObjects[group]
-        groupObjects[group] = [] # release old array
-
     if @tableId
       @calculateHeaderWidths()
 
@@ -384,8 +365,8 @@ class App.ControllerTable extends App.Controller
           do (table, event, callback) ->
             table.delegate('input[name="bulk"]', event, (e) ->
               e.stopPropagation()
-              id      = $(e.target).parents('tr').data('id')
-              checked = $(e.target).prop('checked')
+              id      = $(e.currentTarget).parents('tr').data('id')
+              checked = $(e.currentTarget).prop('checked')
               callback(id, checked, e)
             )
 
@@ -402,29 +383,70 @@ class App.ControllerTable extends App.Controller
     if @checkbox
 
       # click first tr>td, catch click
-      table.delegate('tr > td:nth-child(1)', event, (e) ->
+      table.delegate('tr > td:nth-child(1)', 'click', (e) ->
         e.stopPropagation()
       )
 
       # bind on full bulk click
-      table.delegate('input[name="bulk_all"]', 'change', (e) ->
+      table.delegate('input[name="bulk_all"]', 'change', (e) =>
         e.stopPropagation()
-        if $(e.target).prop('checked')
-          $(e.target).parents('table').find('[name="bulk"]').each( ->
-            if !$(@).prop('checked')
-              #$(@).prop('checked', true)
-              $(@).trigger('click')
+        clicks = []
+        if $(e.currentTarget).prop('checked')
+          $(e.currentTarget).parents('table').find('[name="bulk"]').each( ->
+            $element = $(@)
+            return if $element.prop('checked')
+            $element.prop('checked', true)
+            id = $element.parents('tr').data('id')
+            clicks.push [id, true]
           )
         else
-          $(e.target).parents('table').find('[name="bulk"]').each( ->
-            if $(@).prop('checked')
-              #$(@).prop('checked', false)
-              $(@).trigger('click')
+          $(e.currentTarget).parents('table').find('[name="bulk"]').each( ->
+            $element = $(@)
+            return if !$element.prop('checked')
+            $element.prop('checked', false)
+            id = $element.parents('tr').data('id')
+            clicks.push [id, false]
           )
+        return if !@bindCheckbox
+        return if !@bindCheckbox.events
+        return if _.isEmpty(clicks)
+        for event, callback of @bindCheckbox.events
+          if event == 'click' || event == 'change'
+            for click in clicks
+              callback(click..., e)
       )
-
     table
 
+  sortList: (objects) =>
+
+    for header in @headers
+      if header.name is @orderBy
+        objects = _.sortBy(
+          objects
+          (item) ->
+            # if we need to sort translated col.
+            if header.translate
+              return App.i18n.translateInline(item[header.name])
+
+            # if we need to sort by relation name
+            if header.relation
+              if item[header.name]
+                localItem = App[header.relation].findNative(item[header.name])
+                if localItem && localItem.displayName
+                  localItem = localItem.displayName().toLowerCase()
+                return localItem
+              return ''
+            item[header.name]
+        )
+        if @orderDirection is 'DESC'
+          header.sortOrderIcon = ['arrow-down', 'table-sort-arrow']
+          objects = objects.reverse()
+        else
+          header.sortOrderIcon = ['arrow-up', 'table-sort-arrow']
+      else
+        header.sortOrderIcon = undefined
+    objects
+
   # bind on delete dialog
   deleteRow: (id, e) =>
     e.stopPropagation()

app/assets/javascripts/app/controllers/_integration/ldap.coffee

@@ -24,12 +24,26 @@ class Index extends App.ControllerIntegrationBase
       facility: 'ldap'
     )
 
+  switch: =>
+    super
+    active = @$('.js-switch input').prop('checked')
+    if active
+      @ajax(
+        id:   'jobs_config'
+        type: 'POST'
+        url:  "#{@apiPath}/integration/ldap/job_start"
+        processData: true
+        success: (data, status, xhr) =>
+          @render(true)
+      )
+
 class Form extends App.Controller
   elements:
     '.js-lastImport': 'lastImport'
     '.js-wizard': 'wizardButton'
   events:
     'click .js-wizard': 'startWizard'
+    'click .js-start-sync': 'startSync'
 
   constructor: ->
     super
@@ -42,20 +56,20 @@ class Form extends App.Controller
 
   setConfig: (value) =>
     App.Setting.set('ldap_config', value, {notify: true})
-    @ajax(
-      id:   'jobs_config'
-      type: 'POST'
-      url:  "#{@apiPath}/integration/ldap/job_start"
-      processData: true
-      success: (data, status, xhr) =>
-        @render(true)
-    )
+    @startSync()
 
   render: (top = false) =>
     @config = @currentConfig()
 
+    group_role_map = {}
+    for source, dests of @config.group_role_map
+      group_role_map[source] = dests.map((dest) ->
+        App.Role.find(dest).displayName()
+      ).join ', '
+
     @html App.view('integration/ldap')(
-      config: @config
+      config: @config,
+      group_role_map: group_role_map
     )
     if _.isEmpty(@config)
       @$('.js-notConfigured').removeClass('hide')
@@ -69,6 +83,16 @@ class Form extends App.Controller
         @scrollToIfNeeded($('.content.active .page-header'))
       @delay(a, 500)
 
+  startSync: =>
+    @ajax(
+      id:   'jobs_config'
+      type: 'POST'
+      url:  "#{@apiPath}/integration/ldap/job_start"
+      processData: true
+      success: (data, status, xhr) =>
+        @render(true)
+    )
+
   startWizard: (e) =>
     e.preventDefault()
     new ConnectionWizard(
@@ -152,12 +176,13 @@ class ConnectionWizard extends App.WizardModal
     'click .js-userMappingForm .js-add': 'addUserMapping'
     'click .js-groupRoleForm .js-add':   'addGroupRoleMapping'
     'click .js-goToSlide':               'goToSlide'
-    'input .js-hostUrl':                 'checkSslVerifyDisabled'
+    'input .js-hostUrl':                 'sslVerifyChange'
 
   elements:
     '.modal-body': 'body'
     '.js-userMappingForm': 'userMappingForm'
     '.js-groupRoleForm': 'groupRoleForm'
+    '.js-expertForm': 'expertForm'
 
   constructor: ->
     super
@@ -205,13 +230,27 @@ class ConnectionWizard extends App.WizardModal
 
   showHost: =>
     @$('.js-discover input[name="host_url"]').val(@wizardConfig.host_url)
-    @showSslVerify()
+    @checkSslVerifyVisibility(@wizardConfig.host_url)
+
+  sslVerifyChange: (e) =>
+    @checkSslVerifyVisibility($(e.currentTarget).val())
+
+  checkSslVerifyVisibility: (host_url) =>
+    el     = @$('.js-discover .js-sslVerify')
+    exists = el.length
 
-  showSslVerify: =>
     disabled = true
-    if @wizardConfig.host_url && @wizardConfig.host_url.startsWith('ldaps')
+    if host_url && host_url.startsWith('ldaps')
       disabled = false
 
+    if exists && disabled
+      el.parent().remove()
+    else if !exists && !disabled
+      @$('.js-discover tbody tr').last().after(@buildRowSslVerify())
+
+  buildRowSslVerify: =>
+    el = $(App.view('integration/ldap_ssl_verify_row')())
+
     ssl_verify = true
     if typeof @wizardConfig.ssl_verify != 'undefined'
       ssl_verify = @wizardConfig.ssl_verify
@@ -221,15 +260,11 @@ class ConnectionWizard extends App.WizardModal
       null: false
       options: { true: 'yes', false: 'no' }
       default: ssl_verify
-      disabled: disabled
       translate: true
       class: 'form-control form-control--small'
     )
-    @$('.js-discover .js-sslVerify').html sslVerifyElement
-
-  checkSslVerifyDisabled: (e) =>
-    enabled = $(e.currentTarget).val().startsWith('ldaps')
-    @$('.js-discover .js-sslVerify select[name="ssl_verify"]').prop('disabled', !enabled)
+    el.find('.js-sslVerify').html sslVerifyElement
+    el
 
   discover: (e) =>
     e.preventDefault()
@@ -252,12 +287,13 @@ class ConnectionWizard extends App.WizardModal
 
         option = ''
         options = {}
-        for dn in data.attributes.namingcontexts
-          options[dn] = dn
-          if option is ''
-            option = dn
-          if option.length > dn.length
-            option = dn
+        if !_.isEmpty data.attributes
+          for dn in data.attributes.namingcontexts
+            options[dn] = dn
+            if option is ''
+              option = dn
+            if option.length > dn.length
+              option = dn
 
         @wizardConfig.options = options
         @wizardConfig.option = option
@@ -276,7 +312,7 @@ class ConnectionWizard extends App.WizardModal
 
   bindShow: (alreadyShown) =>
     @showSlide('js-bind') if !alreadyShown
-    @$('.js-bind .js-baseDn').html(@createSelection('base_dn', @wizardConfig.options, @wizardConfig.option))
+    @$('.js-bind .js-baseDn').html(@createSelection('base_dn', @wizardConfig.options, @wizardConfig.base_dn || @wizardConfig.option, true))
     @$('.js-bind input[name="bind_user"]').val(@wizardConfig.bind_user)
     @$('.js-bind input[name="bind_pw"]').val(@wizardConfig.bind_pw)
 
@@ -358,6 +394,14 @@ class ConnectionWizard extends App.WizardModal
     @groupRoleForm.find('tbody tr.js-entry').remove()
     @groupRoleForm.find('tbody tr').before(@buildRowsGroupRole(@wizardConfig.group_role_map))
 
+    @$('.js-mapping input[name="user_filter"]').val(@wizardConfig.user_filter)
+
+    unassigned_users_choices =
+      sigup_roles: App.i18n.translatePlain('Assign signup roles')
+      skip_sync: App.i18n.translatePlain('Don\'t synchronize')
+
+    @$('.js-unassignedUsers').html(@createSelection('unassigned_users', unassigned_users_choices, @wizardConfig.unassigned_users || 'sigup_roles'))
+
   mappingChange: (e) =>
     e.preventDefault()
 
@@ -383,9 +427,16 @@ class ConnectionWizard extends App.WizardModal
     length = group_role_map.source.length-1
     for count in [0..length]
       if group_role_map.source[count] && group_role_map.dest[count]
-        group_role_map_local[group_role_map.source[count]] = group_role_map.dest[count]
+        if !_.isArray(group_role_map_local[group_role_map.source[count]])
+          group_role_map_local[group_role_map.source[count]] = []
+        group_role_map_local[group_role_map.source[count]].push group_role_map.dest[count]
     @wizardConfig.group_role_map = group_role_map_local
 
+    expertSettings = @formParam(@expertForm)
+
+    @wizardConfig.user_filter      = expertSettings.user_filter
+    @wizardConfig.unassigned_users = expertSettings.unassigned_users
+
     @tryShow()
 
   buildRowsUserMap: (user_attribute_map) =>
@@ -413,8 +464,9 @@ class ConnectionWizard extends App.WizardModal
 
   buildRowsGroupRole: (group_role_map) =>
     el = []
-    for source, dest of group_role_map
-      el.push @buildRowGroupRole(source, dest)
+    for source, dests of group_role_map
+      for dest in dests
+        el.push @buildRowGroupRole(source, dest)
     el
 
   buildRowGroupRole: (source, dest) =>
@@ -423,7 +475,7 @@ class ConnectionWizard extends App.WizardModal
     el.find('.js-roleList').html(@createSelection('dest', @wizardConfig.wizardData.roles, dest))
     el
 
-  createSelection: (name, options, selected) ->
+  createSelection: (name, options, selected, unknown) ->
     return App.UiElement.searchable_select.render(
       name: name
       multiple: false
@@ -432,6 +484,7 @@ class ConnectionWizard extends App.WizardModal
       nulloption: false
       options: options
       value: selected
+      unknown: unknown
       class: 'form-control--small'
     )
 
@@ -473,9 +526,9 @@ class ConnectionWizard extends App.WizardModal
         finished: true
       processData: true
       success: (job, status, xhr) =>
-        if job.result && job.result.error
+        if job.result && (job.result.error || job.result.info)
           @showSlide('js-error')
-          @showAlert('js-error', job.result.error)
+          @showAlert('js-error', (job.result.error || job.result.info))
           return
 
         if job.result && job.result.sum

app/assets/javascripts/app/controllers/_ui_element/object_manager_attribute.coffee

@@ -83,7 +83,7 @@ class App.UiElement.object_manager_attribute extends App.UiElement.ApplicationUi
           invite_customer:
             show: false
             required: false
-        'admin.group':
+        'admin.user':
           create:
             shown: true
             required: false
@@ -111,7 +111,7 @@ class App.UiElement.object_manager_attribute extends App.UiElement.ApplicationUi
             required: false
           view:
             shown: true
-        'admin.group':
+        'admin.organization':
           create:
             shown: true
             required: false

app/assets/javascripts/app/controllers/agent_ticket_create.coffee

@@ -228,7 +228,7 @@ class App.TicketCreate extends App.Controller
         type = @$('[name="formSenderType"]').val()
 
         if signature isnt undefined &&  signature.body && type is 'email-out'
-          signatureFinished = App.Utils.replaceTags(signature.body, { user: App.Session.get() })
+          signatureFinished = App.Utils.replaceTags(signature.body, { user: App.Session.get(), config: App.Config.all() })
 
           body = @$('[data-name=body]')
           if App.Utils.signatureCheck(body.html() || '', signatureFinished)
@@ -330,6 +330,9 @@ class App.TicketCreate extends App.Controller
     # show text module UI
     @textModule = new App.WidgetTextModule(
       el: @$('[data-name="body"]').parent()
+      data:
+        config: App.Config.all()
+        user: App.Session.get()
     )
 
     new Sidebar(

app/assets/javascripts/app/controllers/agent_ticket_merge.coffee

@@ -98,9 +98,13 @@ class App.TicketMerge extends App.ControllerModal
             type:    'error'
             msg:     App.i18n.translateContent(data['message'])
             timeout: 6000
-
           @formEnable(e)
 
-      error: =>
+      error: (data) =>
+        details = data.responseJSON || {}
+        @notify
+          type:    'error'
+          msg:     App.i18n.translateContent(details.error_human || details.error || 'Unable to merge!')
+          timeout: 6000
         @formEnable(e)
     )

app/assets/javascripts/app/controllers/import_zendesk.coffee

@@ -10,6 +10,7 @@ class Index extends App.ControllerContent
     '.zendesk-api-token-error':              'apiTokenErrorMessage'
     '#zendesk-email':                        'zendeskEmail'
     '#zendesk-api-token':                    'zendeskApiToken'
+    '.js-ticket-count-info':                 'ticketCountInfo'
   updateMigrationDisplayLoop: 0
 
   events:
@@ -116,7 +117,8 @@ class Index extends App.ControllerContent
   showCredentials: (e) =>
     e.preventDefault()
     @urlStatus.attr('data-state', '')
-    @zendeskUrlApiToken.attr('href', @zendeskUrl.val() + 'agent/admin/api')
+    url = @zendeskUrl.val() + '/agent/admin/api'
+    @zendeskUrlApiToken.attr('href', url.replace(/([^:])\/\/+/g, '$1/'))
     @zendeskUrlApiToken.val('HERE')
     @$('[data-slide=zendesk-url]').toggleClass('hide')
     @$('[data-slide=zendesk-credentials]').toggleClass('hide')
@@ -171,6 +173,10 @@ class Index extends App.ControllerContent
           for key, item of data.data
             if item.done > item.total
               item.done = item.total
+
+            if key == 'Ticket' && item.total >= 1000
+              @ticketCountInfo.removeClass('hide')
+
             element = @$('.js-' + key.toLowerCase() )
             element.find('.js-done').text(item.done)
             element.find('.js-total').text(item.total)

app/assets/javascripts/app/controllers/ticket_overview.coffee

@@ -120,9 +120,18 @@ class App.TicketOverview extends App.Controller
       options:
         duration: 200
         complete: =>
+          if !@hoveredBatchEntry
+            @cleanUpDrag()
+            return
+
           @hoveredBatchEntry.velocity 'reverse',
             duration: 200
             complete: =>
+
+              if !@hoveredBatchEntry
+                @cleanUpDrag()
+                return
+
               # clean scale
               action = @hoveredBatchEntry.attr('data-action')
               id = @hoveredBatchEntry.attr('data-id')
@@ -212,7 +221,7 @@ class App.TicketOverview extends App.Controller
             if @batchCountIndex == @batchCount
               App.Event.trigger('overview:fetch')
         )
-        return
+      return
 
     if action is 'group_assign'
       @batchCount = items.length
@@ -921,7 +930,7 @@ class Table extends App.Controller
     # get ticket list
     ticketListShow = []
     for ticket in tickets
-      ticketListShow.push App.Ticket.fullLocal(ticket.id)
+      ticketListShow.push App.Ticket.find(ticket.id)
 
     # if customer and no ticket exists, show the following message only
     if !ticketListShow[0] && @permissionCheck('ticket.customer')

app/assets/javascripts/app/controllers/ticket_zoom/article_new.coffee

@@ -300,6 +300,7 @@ class App.TicketZoomArticleNew extends App.Controller
         data:
           ticket: ticket
           user: App.Session.get()
+          config: App.Config.all()
       )
       callback = (ticket) ->
         textModule.reload(
@@ -500,7 +501,7 @@ class App.TicketZoomArticleNew extends App.Controller
         @$('[data-name=body] [data-signature="true"]').remove()
 
       # apply new signature
-      signatureFinished = App.Utils.replaceTags(signature.body, { user: App.Session.get(), ticket: ticketCurrent })
+      signatureFinished = App.Utils.replaceTags(signature.body, { user: App.Session.get(), ticket: ticketCurrent, config: App.Config.all() })
 
       body = @$('[data-name=body]')
       if App.Utils.signatureCheck(body.html() || '', signatureFinished)

app/assets/javascripts/app/controllers/ticket_zoom/sidebar_organization.coffee

@@ -1,5 +1,6 @@
 class SidebarOrganization extends App.Controller
   sidebarItem: =>
+    return if !@permissionCheck('ticket.agent')
     return if !@ticket.organization_id
     {
       head: 'Organization'

app/assets/javascripts/app/index.coffee

@@ -195,7 +195,7 @@ class App extends Spine.Controller
 
       # if date is a object, get name of the object
       isObject = false
-      if typeof item is 'object'
+      if item && typeof item is 'object'
         isObject = true
         if item.displayNameLong
           resultLocal = item.displayNameLong()

app/assets/javascripts/app/lib/app_init/session.coffee

@@ -8,12 +8,12 @@ class App.Session
   @get: ( key ) ->
     if _instance == undefined
       _instance ?= new _sessionSingleton
-    _instance.get( key )
+    _instance.get(key)
 
   @set: ( user ) ->
     if _instance == undefined
       _instance ?= new _sessionSingleton
-    _instance.set( user )
+    _instance.set(user)
 
 class _sessionSingleton extends Spine.Module
   @include App.LogInclude

app/assets/javascripts/app/lib/app_post/searchable_select.coffee

@@ -32,6 +32,8 @@ class App.SearchableSelect extends Spine.Controller
     if firstSelected
       @options.attribute.valueName = firstSelected.name
       @options.attribute.value = firstSelected.value
+    else if @options.attribute.unknown && @options.attribute.value
+      @options.attribute.valueName = @options.attribute.value
 
     @options.attribute.renderedOptions = App.view('generic/searchable_select_options')
       options: @options.attribute.options
@@ -142,9 +144,14 @@ class App.SearchableSelect extends Spine.Controller
 
     event.preventDefault()
 
-    @input.val @option_items.filter('.is-active').text().trim()
+    selected = @option_items.filter('.is-active')
+    if selected.length || !@options.attribute.unknown
+      valueName = selected.text().trim()
+      value     = selected.attr('data-value')
+      @input.val valueName
+      @shadowInput.val value
+
     @input.trigger('change')
-    @shadowInput.val @option_items.filter('.is-active').attr('data-value')
     @shadowInput.trigger('change')
     @toggle()
 
@@ -162,6 +169,9 @@ class App.SearchableSelect extends Spine.Controller
     @query = @input.val()
     @filterByQuery @query
 
+    if @options.attribute.unknown
+      @shadowInput.val @query
+
   filterByQuery: (query) ->
     query = escapeRegExp(query)
     regex = new RegExp(query.split(' ').join('.*'), 'i')
@@ -172,7 +182,11 @@ class App.SearchableSelect extends Spine.Controller
         @textContent.match(regex)
       .removeClass 'is-hidden'
 
-    @highlightFirst(true)
+    if @options.attribute.unknown && @option_items.length == @option_items.filter('.is-hidden').length
+      @option_items.removeClass 'is-hidden'
+      @option_items.removeClass 'is-active'
+    else
+      @highlightFirst(true)
 
   highlightFirst: (autocomplete) ->
     first = @option_items.removeClass('is-active').not('.is-hidden').first()

app/assets/javascripts/app/lib/app_post/utils.coffee

@@ -49,10 +49,12 @@ class App.Utils
   @linkify: (string) ->
     window.linkify(string)
 
-  # htmlEscapedAndLinkified = App.Utils.linkify(rawText)
+  # htmlEscapedAndPhoneified = App.Utils.phoneify(rawText)
   @phoneify: (string) ->
-    string = string.replace(/\s+/g, '')
-    "tel://#{encodeURIComponent(string)}"
+    return string if _.isEmpty(string)
+    string = string.replace(/[^0-9,\+,#,\*]+/g, '')
+      .replace(/(.)\+/, '$1')
+    "tel:#{string}"
 
   # wrappedText = App.Utils.wrap(rawText, maxLineLength)
   @wrap: (ascii, max = 82) ->
@@ -125,6 +127,7 @@ class App.Utils
       child = el.firstChild
       break if !child
       break if child.nodeType isnt 1 || child.tagName isnt 'BR'
+      break if !child.remove
       child.remove()
 
     loop
@@ -133,6 +136,7 @@ class App.Utils
       child = el.lastChild
       break if !child
       break if child.nodeType isnt 1 || child.tagName isnt 'BR'
+      break if !child.remove
       child.remove()
 
   # true|false = App.Utils.htmlLastLineEmpty(element)
@@ -535,6 +539,7 @@ class App.Utils
   # textReplaced = App.Utils.replaceTags( template, { user: { firstname: 'Bob', lastname: 'Smith' } } )
   @replaceTags: (template, objects) ->
     template = template.replace( /#\{\s{0,2}(.+?)\s{0,2}\}/g, (index, key) ->
+      key = key.replace(/<.+?>/g, '')
       levels  = key.split(/\./)
       dataRef = objects
       for level in levels

app/assets/javascripts/app/lib/base/jquery.contenteditable.js

@@ -254,6 +254,9 @@
       if (e.clipboardData) { // ie
         clipboardData = e.clipboardData
       }
+      else if (window.clipboardData) { // ie
+        clipboardData = window.clipboardData
+      }
       else if (e.originalEvent.clipboardData) { // other browsers
         clipboardData = e.originalEvent.clipboardData
       }
@@ -292,7 +295,7 @@
               else {
                 img = "<img style=\"width: 100%; max-width: " + width + "px;\" src=\"" + result + "\">"
               }
-              document.execCommand('insertHTML', false, img)
+              _this.paste(img)
             }
 
             // resize if to big
@@ -307,15 +310,23 @@
       }
 
       // check existing + paste text for limit
-      var text = clipboardData.getData('text/html')
-      var docType = 'html'
-      if (!text || text.length === 0) {
-          docType = 'text'
-          text = clipboardData.getData('text/plain')
+      var text, docType
+      try {
+        text = clipboardData.getData('text/html')
+        docType = 'html'
+        if (!text || text.length === 0) {
+            docType = 'text'
+            text = clipboardData.getData('text/plain')
+        }
+        if (!text || text.length === 0) {
+            docType = 'text2'
+            text = clipboardData.getData('text')
+        }
       }
-      if (!text || text.length === 0) {
-          docType = 'text2'
-          text = clipboardData.getData('text')
+      catch (e) {
+        console.log('Sorry, can\'t insert markup because browser is not supporting it.')
+        docType = 'text3'
+        text = clipboardData.getData('text')
       }
       _this.log('paste', docType, text)
 
@@ -355,7 +366,8 @@
       // cleanup
       text = App.Utils.removeEmptyLines(text)
       _this.log('insert', text)
-      document.execCommand('insertHTML', false, text)
+
+      _this.paste(text)
       return true
     })
 
@@ -525,7 +537,30 @@
     }
   }
 
-  $.fn[pluginName] = function ( options ) {
+  // paste some content
+  Plugin.prototype.paste = function(string) {
+    var isIE11 = !!window.MSInputMethodContext && !!document.documentMode;
+
+    // IE <= 10
+    if (document.selection && document.selection.createRange) {
+      var range = document.selection.createRange()
+      if (range.pasteHTML) {
+        range.pasteHTML(string)
+      }
+    }
+    // IE == 11
+    else if (isIE11 && document.getSelection) {
+      var range = document.getSelection().getRangeAt(0)
+      var nnode = document.createElement('div')
+          range.surroundContents(nnode)
+          nnode.innerHTML = string
+    }
+    else {
+      document.execCommand('insertHTML', false, string)
+    }
+  }
+
+  $.fn[pluginName] = function (options) {
     return this.each(function () {
       if (!$.data(this, 'plugin_' + pluginName)) {
         $.data(this, 'plugin_' + pluginName,
@@ -537,6 +572,9 @@
   // get correct val if textbox
   $.fn.ceg = function() {
     var plugin = $.data(this[0], 'plugin_' + pluginName)
+    if (!plugin) {
+      return
+    }
     return plugin.value()
   }
 

app/assets/javascripts/app/lib/base/jquery.textmodule.js

@@ -43,14 +43,17 @@
 
     this.$element.on('keydown', function (e) {
 
-      // esc
-      if (e.keyCode === 27) {
-        _this.close()
-      }
-
       // navigate through item
       if (_this.isActive()) {
 
+        // esc
+        if (e.keyCode === 27) {
+          e.preventDefault()
+          e.stopPropagation()
+          _this.close()
+          return
+        }
+
         // enter
         if (e.keyCode === 13) {
           e.preventDefault()
@@ -72,12 +75,14 @@
         // arrow keys left/right
         if (e.keyCode === 37 || e.keyCode === 39) {
           e.preventDefault()
+          e.stopPropagation()
           return
         }
 
         // up or down
         if (e.keyCode === 38 || e.keyCode === 40) {
           e.preventDefault()
+          e.stopPropagation()
           var active = _this.$widget.find('.dropdown-menu li.is-active')
           active.removeClass('is-active')
 
@@ -102,7 +107,11 @@
             menu.scrollTop( menu.scrollTop() + invisibleHeight )
           }
         }
+      }
 
+      // esc
+      if (e.keyCode === 27) {
+        _this.close()
       }
     })
 
@@ -250,9 +259,21 @@
 
   // paste some content
   Plugin.prototype.paste = function(string) {
-    if (document.selection) { // IE
+    var isIE11 = !!window.MSInputMethodContext && !!document.documentMode;
+
+    // IE <= 10
+    if (document.selection && document.selection.createRange) {
       var range = document.selection.createRange()
-      range.pasteHTML(string)
+      if (range.pasteHTML) {
+        range.pasteHTML(string)
+      }
+    }
+    // IE == 11
+    else if (isIE11 && document.getSelection) {
+      var range = document.getSelection().getRangeAt(0)
+      var nnode = document.createElement('div')
+          range.surroundContents(nnode)
+          nnode.innerHTML = string
     }
     else {
       document.execCommand('insertHTML', false, string)
@@ -295,14 +316,7 @@
     // for chrome, insert space again
     if (start) {
       if (spacerChar === ' ') {
-        string = "&nbsp;"
-        if (document.selection) { // IE
-          var range = document.selection.createRange()
-          range.pasteHTML(string)
-        }
-        else {
-          document.execCommand('insertHTML', false, string)
-        }
+        this.paste('&nbsp;')
       }
     }
   }
@@ -325,7 +339,7 @@
     }
     for (var i = 0; i < this.collection.length; i++) {
       var item = this.collection[i]
-      if ( item.id == id ) {
+      if (item.id == id) {
         var content = item.content
         this.cutInput()
         this.paste(content)

app/assets/javascripts/app/models/overview.coffee

@@ -5,8 +5,8 @@ class App.Overview extends App.Model
   @configure_attributes = [
     { name: 'name',       display: 'Name',                tag: 'input',    type: 'text', limit: 100, 'null': false },
     { name: 'link',       display: 'Link',                readonly: 1 },
-    { name: 'role_ids',   display: 'Available for Role',  tag: 'column_select', multiple: true, nulloption: true, null: false, relation: 'Role', translate: true },
-    { name: 'user_ids',   display: 'Available for User',  tag: 'column_select', multiple: true, nulloption: false, null: true,  relation: 'User', sortBy: 'firstname' },
+    { name: 'role_ids',   display: 'Available for Role',  tag: 'column_select', multiple: true, null: false, relation: 'Role', translate: true },
+    { name: 'user_ids',   display: 'Available for User',  tag: 'column_select', multiple: true, null: true,  relation: 'User', sortBy: 'firstname' },
     { name: 'organization_shared', display: 'Only available for Users with shared Organization', tag: 'select', options: { true: 'yes', false: 'no' }, default: false, null: true },
     { name: 'condition',  display: 'Conditions for shown Tickets', tag: 'ticket_selector', null: false },
     { name: 'prio',       display: 'Prio',                readonly: 1 },

app/assets/javascripts/app/views/channel/email_notification_wizard.jst.eco

@@ -35,6 +35,7 @@
   </form>
 
   <form class="modal-content setup wizard js-outbound">
+    <div class="modal-header">
       <div class="modal-close js-close">
         <%- @Icon('diagonal-cross') %>
       </div>

app/assets/javascripts/app/views/import/zendesk.jst.eco

@@ -61,6 +61,7 @@
     <div class="wizard-slide vertical hide" data-slide="zendesk-import">
       <h2><%- @T('%s Migration', 'Zendesk') %></h2>
       <div class="alert alert--danger hide js-error" role="alert"></div>
+      <div class="alert alert--info hide js-ticket-count-info" role="alert"><%- @T("There are more than 1000 tickets in the Zendesk system. Due to API rate limit restrictions we can't get the exact number of tickets yet and have to fetch them in batches of 1000. This might take some time, better grab a cup of coffee. The total number of tickets gets updated as soon as the currently known number is surpassed.") %></div>
       <div class="wizard-body flex vertical justified">
         <table class="progressTable">
           <tr class="js-group">

app/assets/javascripts/app/views/integration/ldap.jst.eco

@@ -64,7 +64,7 @@
   <% end %>
 
   <h3><%- @T('Role') %></h3>
-  <% if _.isEmpty(@config.group_role_map): %>
+  <% if _.isEmpty(@group_role_map): %>
     <table class="settings-list settings-list--stretch settings-list--placeholder">
       <thead><tr><th><%- @T('No Entries') %>
     </table>
@@ -75,10 +75,10 @@
           <th width="40%"><%- @T('LDAP') %>
           <th width="60%"><%- @T('Zammad') %>
       <tbody>
-        <% for key, value of @config.group_role_map: %>
+        <% for source, dests of @group_role_map: %>
           <tr>
-            <td class="settings-list-row-control"><%= key %>
-            <td class="settings-list-row-control"><%= App.Role.find(value).displayName() %>
+            <td class="settings-list-row-control"><%= source %>
+            <td class="settings-list-row-control"><%= dests %>
         <% end %>
   <% end %>
     </table>

app/assets/javascripts/app/views/integration/ldap_last_import.jst.eco

@@ -3,6 +3,8 @@
   <% if _.isEmpty(@job.started_at): %>
     <% if @job.result && @job.result.error: %>
         <div class="alert alert--danger" role="alert"><%- @T('An error occurred: %s', @job.result.error) %></div>
+    <% else if @job.result && @job.result.info: %>
+        <div class="alert alert--info" role="alert"><%- @T('Info: %s', @job.result.info) %></div>
     <% else: %>
       <p><%- @T('Job is waiting to get started...') %></p>
     <% end %>
@@ -11,6 +13,8 @@
       <p><%- @Ttimestamp(@job.started_at) %> - <%- @Ttimestamp(@job.finished_at) %></p>
       <% if @job.result && @job.result.error: %>
         <div class="alert alert--danger" role="alert"><%- @T('An error occurred: %s', @job.result.error) %></div>
+      <% else if @job.result && @job.result.info: %>
+        <div class="alert alert--info" role="alert"><%- @T('Info: %s', @job.result.info) %></div>
       <% end %>
     <% else: %>
       <% if @job.result && @job.result.error: %>
@@ -25,21 +29,24 @@
         </div>
       <% end %>
     <% end %>
-    <% if !_.isEmpty(@job.result): %>
+    <% if !_.isEmpty(@job.result) && @countDone: %>
       <ul>
         <li><%- @T('%s user to %s user', 'LDAP', 'Zammad') %> (<%= @countDone %>/<%= @job.result.sum %>):
           <ul>
-            <li><%- @T('Users') %>: <%= @job.result.created %> <%- @T('created') %>, <%= @job.result.updated %> <%- @T('updated') %>, <%= @job.result.unchanged %> <%- @T('untouched') %>, <%= @job.result.skipped %> <%- @T('skipped') %>, <%= @job.result.failed %> <%- @T('failed') %>
+            <li><%- @T('Users') %>: <%= @job.result.created %> <%- @T('created') %>, <%= @job.result.updated %> <%- @T('updated') %>, <%= @job.result.unchanged %> <%- @T('untouched') %>, <%= @job.result.skipped %> <%- @T('skipped') %>, <%= @job.result.failed %> <%- @T('failed') %>, <%= @job.result.deactivated %> <%- @T('deactivated') %>
           </ul>
         <% if !_.isEmpty(@job.result.roles): %>
           <li><%- @T('%s groups to %s roles assignments', 'LDAP', 'Zammad') %>:
             <ul>
               <% for role, result of @job.result.roles: %>
-                <li> <%- @T(role) %>: <%= result.created %> <%- @T('created') %>, <%= result.updated %> <%- @T('updated') %>, <%= result.unchanged %> <%- @T('untouched') %>, <%= result.failed %> <%- @T('failed') %>
+                <li> <%- @T(role) %>: <%= result.created %> <%- @T('created') %>, <%= result.updated %> <%- @T('updated') %>, <%= result.unchanged %> <%- @T('untouched') %>, <%= result.failed %> <%- @T('failed') %>, <%= result.deactivated %> <%- @T('deactivated') %>
               <% end %>
             </ul>
         <% end %>
       </ul>
     <% end %>
+    <% if @job.finished_at: %>
+      <button type="submit" class="btn btn--primary js-start-sync"><%- @T('Start new') %></button>
+    <% end %>
   <% end %>
 </div>

app/assets/javascripts/app/views/integration/ldap_ssl_verify_row.jst.eco

@@ -0,0 +1,3 @@
+<tr>
+  <td class="settings-list-row-control"><%- @T('SSL verification') %>
+  <td class="settings-list-control-cell js-sslVerify">

app/assets/javascripts/app/views/integration/ldap_summary.jst.eco

@@ -1,14 +1,14 @@
 <ul>
   <li><%- @T('%s user to %s user', 'LDAP', 'Zammad') %> (<%= @countDone %>):
     <ul>
-      <li><%- @T('Users') %>: <%= @job.result.created %> <%- @T('created') %>, <%= @job.result.updated %> <%- @T('updated') %>, <%= @job.result.unchanged %> <%- @T('untouched') %>, <%= @job.result.skipped %> <%- @T('skipped') %>, <%= @job.result.failed %> <%- @T('failed') %>
+      <li><%- @T('Users') %>: <%= @job.result.created %> <%- @T('created') %>, <%= @job.result.updated %> <%- @T('updated') %>, <%= @job.result.unchanged %> <%- @T('untouched') %>, <%= @job.result.skipped %> <%- @T('skipped') %>, <%= @job.result.failed %> <%- @T('failed') %>, <%= @job.result.deactivated %> <%- @T('deactivated') %>
     </ul>
   </li>
   <% if !_.isEmpty(@job.result.roles): %>
     <li><%- @T('%s groups to %s roles assignments', 'LDAP', 'Zammad') %>:
       <ul>
         <% for role, result of @job.result.roles: %>
-          <li><%- @T(role) %>: <%= result.created %> <%- @T('created') %>, <%= result.updated %> <%- @T('updated') %>, <%= result.unchanged %> <%- @T('untouched') %>, <%= result.failed %> <%- @T('failed') %>
+          <li><%- @T(role) %>: <%= result.created %> <%- @T('created') %>, <%= result.updated %> <%- @T('updated') %>, <%= result.unchanged %> <%- @T('untouched') %>, <%= result.failed %> <%- @T('failed') %>, <%= result.deactivated %> <%- @T('deactivated') %>
         <% end %>
       </ul>
     </li>

app/assets/javascripts/app/views/integration/ldap_wizard.jst.eco

@@ -20,9 +20,6 @@
             <tr>
               <td class="settings-list-row-control"><%- @T('Host') %>
               <td class="settings-list-control-cell"><input type="text" name="host_url" class="form-control form-control--small js-hostUrl" value="" placeholder="ldaps://ldap.example.com" autocomplete="new-password">
-            <tr>
-              <td class="settings-list-row-control"><%- @T('SSL verification') %>
-              <td class="settings-list-control-cell js-sslVerify">
           </tbody>
         </table>
       </div>
@@ -172,7 +169,6 @@
         </form>
 
         <h2><%- @T('Roles') %></h2>
-        <p><%- @T('Note: All not mapped users will get the default signup roles.') %></p>
         <form class="js-groupRoleForm">
           <table class="settings-list js-groupRoleMap" style="width: 100%;">
             <colgroup>
@@ -196,6 +192,25 @@
           </table>
         </form>
 
+        <h2><%- @T('Expert') %></h2>
+        <form class="js-expertForm">
+          <table class="settings-list js-expert" style="width: 100%;">
+            <thead>
+              <tr>
+                <th width="30%"><%- @T('Name') %>
+                <th width="70%"><%- @T('Value') %>
+            </thead>
+            <tbody>
+              <tr>
+                <td class="settings-list-row-control"><%- @T('User filter') %>
+                <td class="settings-list-control-cell"><input type="text" name="user_filter" class="form-control form-control--small" value="" placeholder="" autocomplete="new-password">
+              <tr>
+                <td class="settings-list-row-control"><%- @T('Users without assigned LDAP groups') %>
+                <td class="settings-list-control-cell js-unassignedUsers">
+            </tbody>
+          </table>
+        </form>
+
       </div>
     </div>
     <div class="modal-footer">

app/controllers/form_controller.rb

@@ -5,7 +5,7 @@ class FormController < ApplicationController
   before_action :cors_preflight_check_execute
   after_action :set_access_control_headers_execute
 
-  def config
+  def configuration
     return if !enabled?
 
     api_path  = Rails.configuration.api_path
@@ -14,16 +14,16 @@ class FormController < ApplicationController
 
     endpoint = "#{http_type}://#{fqdn}#{api_path}/form_submit"
 
-    config = {
+    result = {
       enabled:  Setting.get('form_ticket_create'),
       endpoint: endpoint,
     }
 
     if params[:test] && current_user && current_user.permissions?('admin.channel_formular')
-      config[:enabled] = true
+      result[:enabled] = true
     end
 
-    render json: config, status: :ok
+    render json: result, status: :ok
   end
 
   def submit
@@ -112,11 +112,12 @@ class FormController < ApplicationController
     )
 
     if params[:file]
+
       params[:file].each { |file|
         Store.add(
           object: 'Ticket::Article',
           o_id: article.id,
-          data: File.read(file.tempfile),
+          data: file.read,
           filename: file.original_filename,
           preferences: {
             'Mime-Type' => file.content_type,

app/controllers/import_zendesk_controller.rb

@@ -7,7 +7,7 @@ class ImportZendeskController < ApplicationController
     return if setup_done_response
 
     # validate
-    if !params[:url] || params[:url] !~ %r{^(http|https)://.+?$}
+    if params[:url].blank? || params[:url] !~ %r{^(http|https)://.+?$}
       render json: {
         result: 'invalid',
         message: 'Invalid URL!',
@@ -40,8 +40,8 @@ class ImportZendeskController < ApplicationController
       return
     end
 
-    # since 2016-10-15 a redirect to a signup page has been implemented
-    if response.body =~ /(Take it for a risk-free|Take it for a risk-free)/i
+    # since 2016-10-15 a redirect to a marketing page has been implemented
+    if response.body !~ /#{params[:url]}/
       render json: {
         result: 'invalid',
         message_human: 'Hostname not found!',
@@ -49,7 +49,9 @@ class ImportZendeskController < ApplicationController
       return
     end
 
-    Setting.set('import_zendesk_endpoint', "#{params[:url]}api/v2")
+    endpoint = "#{params[:url]}/api/v2"
+    endpoint.gsub!(%r{([^:])//+}, '\\1/')
+    Setting.set('import_zendesk_endpoint', endpoint)
 
     render json: {
       result: 'ok',
@@ -92,6 +94,7 @@ class ImportZendeskController < ApplicationController
   def import_start
     return if setup_done_response
     Setting.set('import_mode', true)
+    Setting.set('import_backend', 'zendesk')
 
     # start migration
     Import::Zendesk.delay.start_bg

app/controllers/integration/ldap_controller.rb

@@ -14,12 +14,21 @@ class Integration::LdapController < ApplicationController
       attributes: ldap.preferences,
     }
   rescue => e
-    logger.error e
+    # workaround for issue #1114
+    if e.message.end_with?(', 48, Inappropriate Authentication')
+      result = {
+        result:     'ok',
+        attributes: {},
+      }
+    else
+      logger.error e
+      result = {
+        result:  'failed',
+        message: e.message,
+      }
+    end
 
-    render json: {
-      result:  'failed',
-      message: e.message,
-    }
+    render json: result
   end
 
   def bind
@@ -71,8 +80,11 @@ class Integration::LdapController < ApplicationController
   end
 
   def job_start_create
-    job = ImportJob.create(name: 'Import::Ldap', payload: Setting.get('ldap_config'))
-    job.delay.start
+    backend = 'Import::Ldap'
+    if !ImportJob.exists?(name: backend, finished_at: nil)
+      job = ImportJob.create(name: backend, payload: Setting.get('ldap_config'))
+      job.delay.start
+    end
     render json: {
       result: 'ok',
     }

app/controllers/monitoring_controller.rb

@@ -32,8 +32,10 @@ curl http://localhost/api/v1/monitoring/health_check?token=XXX
     issues = []
 
     # channel check
+    last_run_tolerance = Time.zone.now - 1.hour
     Channel.where(active: true).each { |channel|
-      next if (channel.status_in.empty? || channel.status_in == 'ok') && (channel.status_out.empty? || channel.status_out == 'ok')
+
+      # inbound channel
       if channel.status_in == 'error'
         message = "Channel: #{channel.area} in "
         %w(host user uid).each { |key|
@@ -42,6 +44,11 @@ curl http://localhost/api/v1/monitoring/health_check?token=XXX
         }
         issues.push "#{message} #{channel.last_log_in}"
       end
+      if channel.preferences && channel.preferences['last_fetch'] && channel.preferences['last_fetch'] < last_run_tolerance
+        issues.push "#{message} channel is active but not fetched for 1 hour"
+      end
+
+      # outbound channel
       next if channel.status_out != 'error'
       message = "Channel: #{channel.area} out "
       %w(host user uid).each { |key|

app/controllers/object_manager_attributes_controller.rb

@@ -1,7 +1,7 @@
 # Copyright (C) 2012-2016 Zammad Foundation, http://zammad-foundation.org/
 
 class ObjectManagerAttributesController < ApplicationController
-  before_action { authentication_check(permission: 'admin.object') }
+  prepend_before_action { authentication_check(permission: 'admin.object') }
 
   # GET /object_manager_attributes_list
   def list

app/controllers/reports_controller.rb

@@ -143,7 +143,7 @@ class ReportsController < ApplicationController
       stop = "#{date}T23:59:59Z"
       range = 'hour'
     elsif params[:timeRange] == 'week'
-      start = Date.commercial(params[:year], params[:week]).iso8601
+      start = Date.commercial(params[:year].to_i, params[:week].to_i).iso8601
       stop = Date.parse(start).end_of_week.iso8601
       range = 'week'
     elsif params[:timeRange] == 'month'
@@ -180,7 +180,7 @@ class ReportsController < ApplicationController
     worksheet.set_column(4, 8, 20)
 
     # Add and define a format
-    format = workbook.add_format  # Add a format
+    format = workbook.add_format
     format.set_bold
     format.set_size(14)
     format.set_color('black')
@@ -189,22 +189,28 @@ class ReportsController < ApplicationController
     # Write a formatted and unformatted string, row and column notation.
     worksheet.write(0, 0, "Tickets: #{profile.name} (#{title})", format)
 
-    format_header = workbook.add_format  # Add a format
+    format_header = workbook.add_format
     format_header.set_italic
     format_header.set_bg_color('gray')
     format_header.set_color('white')
+
     worksheet.write(2, 0, '#', format_header)
     worksheet.write(2, 1, 'Title', format_header)
     worksheet.write(2, 2, 'State', format_header)
     worksheet.write(2, 3, 'Priority', format_header)
     worksheet.write(2, 4, 'Group', format_header)
-    worksheet.write(2, 5, 'Customer', format_header)
-    worksheet.write(2, 6, 'Created at', format_header)
-    worksheet.write(2, 7, 'Updated at', format_header)
-    worksheet.write(2, 8, 'Closed at', format_header)
+    worksheet.write(2, 5, 'Owner', format_header)
+    worksheet.write(2, 6, 'Customer', format_header)
+    worksheet.write(2, 7, 'Organization', format_header)
+    worksheet.write(2, 8, 'Create Channel', format_header)
+    worksheet.write(2, 9, 'Sender', format_header)
+    worksheet.write(2, 10, 'Tags', format_header)
+    worksheet.write(2, 11, 'Created at', format_header)
+    worksheet.write(2, 12, 'Updated at', format_header)
+    worksheet.write(2, 13, 'Closed at', format_header)
 
     row = 2
-    result[:ticket_ids].each { |ticket_id|
+    result[:ticket_ids].each do |ticket_id|
       ticket = Ticket.lookup(id: ticket_id)
       row += 1
       worksheet.write(row, 0, ticket.number)
@@ -212,11 +218,16 @@ class ReportsController < ApplicationController
       worksheet.write(row, 2, ticket.state.name)
       worksheet.write(row, 3, ticket.priority.name)
       worksheet.write(row, 4, ticket.group.name)
-      worksheet.write(row, 5, ticket.customer.fullname)
-      worksheet.write(row, 6, ticket.created_at)
-      worksheet.write(row, 7, ticket.updated_at)
-      worksheet.write(row, 8, ticket.close_at)
-    }
+      worksheet.write(row, 5, ticket.owner.fullname)
+      worksheet.write(row, 6, ticket.customer.fullname)
+      worksheet.write(row, 7, ticket.try(:organization).try(:name))
+      worksheet.write(row, 8, ticket.create_article_type.name)
+      worksheet.write(row, 9, ticket.create_article_sender.name)
+      worksheet.write(row, 10, ticket.tag_list.join(','))
+      worksheet.write(row, 11, ticket.created_at)
+      worksheet.write(row, 12, ticket.updated_at)
+      worksheet.write(row, 13, ticket.close_at)
+    end
 
     workbook.close
 

app/controllers/search_controller.rb

@@ -62,6 +62,7 @@ class SearchController < ApplicationController
         items.each { |item|
           require item[:type].to_filename
           record = Kernel.const_get(item[:type]).lookup(id: item[:id])
+          next if !record
           assets = record.assets(assets)
           result.push item
         }

app/controllers/sessions_controller.rb

@@ -282,12 +282,12 @@ class SessionsController < ApplicationController
     assets = {}
     sessions_clean = []
     SessionHelper.list.each { |session|
-      next if !session.data['user_id']
+      next if session.data['user_id'].blank?
       sessions_clean.push session
-      if session.data['user_id']
-        user = User.lookup(id: session.data['user_id'])
-        assets = user.assets(assets)
-      end
+      next if session.data['user_id']
+      user = User.lookup(id: session.data['user_id'])
+      next if !user
+      assets = user.assets(assets)
     }
     render json: {
       sessions: sessions_clean,

app/controllers/tickets_controller.rb

@@ -345,15 +345,6 @@ class TicketsController < ApplicationController
     # permission check
     ticket_permission(ticket_slave)
 
-    # check diffetent ticket ids
-    if ticket_slave.id == ticket_master.id
-      render json: {
-        result: 'failed',
-        message: 'Can\'t merge ticket with it self!',
-      }
-      return
-    end
-
     # merge ticket
     ticket_slave.merge_to(
       ticket_id: ticket_master.id,

app/controllers/time_accountings_controller.rb

@@ -8,7 +8,7 @@ class TimeAccountingsController < ApplicationController
     year = params[:year] || Time.zone.now.year
     month = params[:month] || Time.zone.now.month
 
-    start_periode = Date.parse("#{year}-#{month}-01")
+    start_periode = Time.zone.parse("#{year}-#{month}-01")
     end_periode = start_periode.end_of_month
 
     time_unit = {}
@@ -125,7 +125,7 @@ class TimeAccountingsController < ApplicationController
     year = params[:year] || Time.zone.now.year
     month = params[:month] || Time.zone.now.month
 
-    start_periode = Date.parse("#{year}-#{month}-01")
+    start_periode = Time.zone.parse("#{year}-#{month}-01")
     end_periode = start_periode.end_of_month
 
     time_unit = {}
@@ -205,7 +205,7 @@ class TimeAccountingsController < ApplicationController
     year = params[:year] || Time.zone.now.year
     month = params[:month] || Time.zone.now.month
 
-    start_periode = Date.parse("#{year}-#{month}-01")
+    start_periode = Time.zone.parse("#{year}-#{month}-01")
     end_periode = start_periode.end_of_month
 
     time_unit = {}

app/models/application_model.rb

@@ -5,7 +5,7 @@ class ApplicationModel < ActiveRecord::Base
   include ApplicationModel::Cacheable
   include ApplicationModel::CanLookup
   include ApplicationModel::CanLookupSearchIndexAttributes
-  include ApplicationModel::ChecksAttributeLength
+  include ApplicationModel::ChecksAttributeValuesAndLength
   include ApplicationModel::CleansParam
   include ApplicationModel::CleansRecentView
   include ApplicationModel::FillsByUserColumns

app/models/application_model/checks_attribute_length.rb

@@ -1,38 +0,0 @@
-# Copyright (C) 2012-2016 Zammad Foundation, http://zammad-foundation.org/
-module ApplicationModel::ChecksAttributeLength
-  extend ActiveSupport::Concern
-
-  included do
-    before_create :check_attribute_length
-    before_update :check_attribute_length
-  end
-
-=begin
-
-check string/varchar size and cut them if needed
-
-=end
-
-  def check_attribute_length
-    attributes.each { |attribute|
-      next if !self[ attribute[0] ]
-      next if !self[ attribute[0] ].instance_of?(String)
-      next if self[ attribute[0] ].empty?
-      column = self.class.columns_hash[ attribute[0] ]
-      next if !column
-      limit = column.limit
-      if column && limit
-        current_length = attribute[1].to_s.length
-        if limit < current_length
-          logger.warn "WARNING: cut string because of database length #{self.class}.#{attribute[0]}(#{limit} but is #{current_length}:#{attribute[1]})"
-          self[ attribute[0] ] = attribute[1][ 0, limit ]
-        end
-      end
-
-      # strip 4 bytes utf8 chars if needed
-      if column && self[ attribute[0] ]
-        self[attribute[0]] = self[ attribute[0] ].utf8_to_3bytesutf8
-      end
-    }
-  end
-end

app/models/application_model/checks_attribute_values_and_length.rb

@@ -0,0 +1,54 @@
+# Copyright (C) 2012-2016 Zammad Foundation, http://zammad-foundation.org/
+module ApplicationModel::ChecksAttributeValuesAndLength
+  extend ActiveSupport::Concern
+
+  included do
+    before_create :check_attribute_values_and_length
+    before_update :check_attribute_values_and_length
+  end
+
+=begin
+
+1) check string/varchar size and cut them if needed
+
+2) check string for null byte \u0000 and remove it
+
+=end
+
+  def check_attribute_values_and_length
+    columns = self.class.columns_hash
+    attributes.each { |name, value|
+      next if value.blank?
+      next if !value.instance_of?(String)
+      column = columns[name]
+      next if !column
+
+      # strip null byte chars (postgresql will complain about it)
+      if column.sql_type == 'text'
+        if Rails.application.config.db_null_byte == false
+          self[name].delete!("\u0000")
+        end
+      end
+
+      # for varchar check length and replace null bytes
+      limit = column.limit
+      if limit
+        current_length = value.length
+        if limit < current_length
+          logger.warn "WARNING: cut string because of database length #{self.class}.#{name}(#{limit} but is #{current_length}:#{value})"
+          self[name] = value[0, limit]
+        end
+
+        # strip null byte chars (postgresql will complain about it)
+        if Rails.application.config.db_null_byte == false
+          self[name].delete!("\u0000")
+        end
+      end
+
+      # strip 4 bytes utf8 chars if needed (mysql/mariadb will complain it)
+      next if self[name].blank?
+      self[name] = self[name].utf8_to_3bytesutf8
+    }
+    true
+  end
+end

app/models/channel.rb

@@ -132,45 +132,57 @@ stream all accounts
   def self.stream
     Thread.abort_on_exception = true
 
+    auto_reconnect_after = 25
     last_channels = []
 
     loop do
       logger.debug 'stream controll loop'
+
       current_channels = []
       channels = Channel.where('active = ? AND area LIKE ?', true, '%::Account')
       channels.each { |channel|
         next if channel.options[:adapter] != 'twitter'
+        channel_id = channel.id.to_s
+        current_channels.push channel_id
 
-        current_channels.push channel.id
-
-        # exit it channel has changed
-        if @@channel_stream[channel.id] && @@channel_stream[channel.id][:updated_at] != channel.updated_at
-          logger.debug "channel (#{channel.id}) has changed, restart thread"
-          @@channel_stream[channel.id][:thread].exit
-          @@channel_stream[channel.id][:thread].join
-          @@channel_stream[channel.id][:stream_instance].disconnect
-          @@channel_stream[channel.id] = false
+        # exit it channel has changed or connection is older then 25 min.
+        if @@channel_stream[channel_id]
+          if @@channel_stream[channel_id][:updated_at] != channel.updated_at
+            logger.info "channel (#{channel.id}) has changed, stop thread"
+            @@channel_stream[channel_id][:thread].exit
+            @@channel_stream[channel_id][:thread].join
+            @@channel_stream[channel_id][:stream_instance].disconnect
+            @@channel_stream[channel_id] = false
+          elsif @@channel_stream[channel_id][:started_at] && @@channel_stream[channel_id][:started_at] < Time.zone.now - auto_reconnect_after.minutes
+            logger.info "channel (#{channel.id}) reconnect - thread is older then #{auto_reconnect_after} minutes, restart thread"
+            @@channel_stream[channel_id][:thread].exit
+            @@channel_stream[channel_id][:thread].join
+            @@channel_stream[channel_id][:stream_instance].disconnect
+            @@channel_stream[channel_id] = false
+          end
         end
 
-        #logger.debug "thread for channel (#{channel.id}) already running" if @@channel_stream[channel.id]
-        next if @@channel_stream[channel.id]
+        #logger.debug "thread for channel (#{channel.id}) already running" if channel_stream
+        next if @@channel_stream[channel_id]
 
-        @@channel_stream[channel.id] = {
-          updated_at: channel.updated_at
+        @@channel_stream[channel_id] = {
+          updated_at: channel.updated_at,
+          started_at: Time.zone.now,
         }
 
         # start channels with delay
         sleep @@channel_stream.count
 
         # start threads for each channel
-        @@channel_stream[channel.id][:thread] = Thread.new {
+        @@channel_stream[channel_id][:thread] = Thread.new {
           begin
             logger.info "Started stream channel for '#{channel.id}' (#{channel.area})..."
-            @@channel_stream[channel.id][:stream_instance] = channel.stream_instance
-            @@channel_stream[channel.id][:stream_instance].stream
-            @@channel_stream[channel.id][:stream_instance].disconnect
-            @@channel_stream[channel.id] = false
-            logger.debug " ...stopped thread for '#{channel.id}'"
+            @@channel_stream[channel_id] ||= {}
+            @@channel_stream[channel_id][:stream_instance] = channel.stream_instance
+            @@channel_stream[channel_id][:stream_instance].stream
+            @@channel_stream[channel_id][:stream_instance].disconnect
+            @@channel_stream[channel_id] = false
+            logger.info " ...stopped thread for '#{channel.id}'"
           rescue => e
             error = "Can't use channel (#{channel.id}): #{e.inspect}"
             logger.error error
@@ -178,24 +190,24 @@ stream all accounts
             channel.status_in = 'error'
             channel.last_log_in = error
             channel.save
-            @@channel_stream[channel.id] = false
+            @@channel_stream[channel_id] = false
           end
         }
       }
 
       # cleanup deleted channels
       last_channels.each { |channel_id|
-        next if !@@channel_stream[channel_id]
+        next if !@@channel_stream[channel_id.to_s]
         next if current_channels.include?(channel_id)
-        logger.debug "channel (#{channel_id}) not longer active, stop thread"
-        @@channel_stream[channel_id][:thread].exit
-        @@channel_stream[channel_id][:thread].join
-        @@channel_stream[channel_id][:stream_instance].disconnect
-        @@channel_stream[channel_id] = false
+        logger.info "channel (#{channel_id}) not longer active, stop thread"
+        @@channel_stream[channel_id.to_s][:thread].exit
+        @@channel_stream[channel_id.to_s][:thread].join
+        @@channel_stream[channel_id.to_s][:stream_instance].disconnect
+        @@channel_stream[channel_id.to_s] = false
       }
       last_channels = current_channels
 
-      sleep 30
+      sleep 20
     end
 
   end

app/models/channel/driver/twitter.rb

@@ -82,7 +82,7 @@ returns
     # only fetch once in 30 minutes
     return true if !channel.preferences
     return true if !channel.preferences[:last_fetch]
-    return false if channel.preferences[:last_fetch] > Time.zone.now - 30.minutes
+    return false if channel.preferences[:last_fetch] > Time.zone.now - 20.minutes
     true
   end
 
@@ -183,6 +183,24 @@ returns
 =end
 
   def stream
+    sleep_on_unauthorized = 61
+    2.times { |loop_count|
+      begin
+        stream_start
+      rescue Twitter::Error::Unauthorized => e
+        Rails.logger.info "Unable to stream, try #{loop_count}, error #{e.inspect}"
+        if loop_count < 2
+          Rails.logger.info "wait for #{sleep_on_unauthorized} sec. and try it again"
+          sleep sleep_on_unauthorized
+        else
+          raise "Unable to stream, try #{loop_count}, error #{e.inspect}"
+        end
+      end
+    }
+  end
+
+  def stream_start
+
     sync = @channel.options['sync']
     raise 'Need channel.options[\'sync\'] for account, but no params found' if !sync
 
@@ -204,20 +222,21 @@ returns
       next if tweet.class != Twitter::Tweet && tweet.class != Twitter::DirectMessage
 
       # wait until own posts are stored in local database to prevent importing own tweets
-      sleep 4
+      next if @stream_client.locale_sender?(tweet) && own_tweet_already_imported?(tweet)
+
       next if Ticket::Article.find_by(message_id: tweet.id)
 
       # check direct message
       if tweet.class == Twitter::DirectMessage
         if sync['direct_messages'] && sync['direct_messages']['group_id'] != ''
-          next if @stream_client.direct_message_limit_reached(tweet)
+          next if @stream_client.direct_message_limit_reached(tweet, 2)
           @stream_client.to_group(tweet, sync['direct_messages']['group_id'], @channel)
         end
         next
       end
 
       next if !track_retweets? && tweet.retweet?
-      next if @stream_client.tweet_limit_reached(tweet)
+      next if @stream_client.tweet_limit_reached(tweet, 2)
 
       # check if it's mention
       if sync['mentions'] && sync['mentions']['group_id'] != ''
@@ -290,6 +309,9 @@ returns
           Rails.logger.debug "tweet to old: #{tweet.id}/#{tweet.created_at}"
           next
         end
+
+        next if @rest_client.locale_sender?(tweet) && own_tweet_already_imported?(tweet)
+
         next if Ticket::Article.find_by(message_id: tweet.id)
         break if @rest_client.tweet_limit_reached(tweet)
         @rest_client.to_group(tweet, search[:group_id], @channel)
@@ -324,7 +346,7 @@ returns
     Rails.logger.debug ' - searching for direct_messages'
     older_import = 0
     older_import_max = 20
-    @rest_client.client.direct_messages.each { |tweet|
+    @rest_client.client.direct_messages(full_text: 'true').each { |tweet|
 
       # ignore older messages
       if (@channel.created_at - 15.days) > tweet.created_at || older_import >= older_import_max
@@ -351,4 +373,28 @@ returns
   def track_retweets?
     @channel.options && @channel.options['sync'] && @channel.options['sync']['track_retweets']
   end
+
+  def own_tweet_already_imported?(tweet)
+    event_time = Time.zone.now
+    sleep 4
+    12.times { |loop_count|
+      if Ticket::Article.find_by(message_id: tweet.id)
+        Rails.logger.debug "Own tweet already imported, skipping tweet #{tweet.id}"
+        return true
+      end
+      count = Delayed::Job.where('created_at < ?', event_time).count
+      break if count.zero?
+      sleep_time = 2 * count
+      sleep_time = 5 if sleep_time > 5
+      Rails.logger.debug "Delay importing own tweets - sleep #{sleep_time} (loop #{loop_count})"
+      sleep sleep_time
+    }
+
+    if Ticket::Article.find_by(message_id: tweet.id)
+      Rails.logger.debug "Own tweet already imported, skipping tweet #{tweet.id}"
+      return true
+    end
+    false
+  end
+
 end

app/models/channel/email_build.rb

@@ -156,7 +156,9 @@ Check if string is a complete html document. If not, add head and css styles.
 
     return html if html =~ /<html>/i
 
-    Rails.configuration.html_email_body.sub('###html###', html)
+    # use block form because variable html could contain backslashes and e. g. '\1' that
+    # must not be handled as back-references for regular expressions
+    Rails.configuration.html_email_body.sub('###html###') { html }
   end
 
 =begin

app/models/channel/email_parser.rb

@@ -98,14 +98,24 @@ class Channel::EmailParser
       data[field.to_sym] = ''
     }
 
-    # get sender
+    # get sender with @ / email address
     from = nil
     ['from', 'reply-to', 'return-path'].each { |item|
       next if data[item.to_sym].blank?
+      next if data[item.to_sym] !~ /@/
       from = data[item.to_sym]
       break if from
     }
 
+    # in case of no sender with email address - get sender
+    if !from
+      ['from', 'reply-to', 'return-path'].each { |item|
+        next if data[item.to_sym].blank?
+        from = data[item.to_sym]
+        break if from
+      }
+    end
+
     # set x-any-recipient
     data['x-any-recipient'.to_sym] = ''
     ['to', 'cc', 'delivered-to', 'x-original-to', 'envelope-to'].each { |item|
@@ -117,32 +127,7 @@ class Channel::EmailParser
     }
 
     # set extra headers
-    begin
-      data[:from_email]        = Mail::Address.new(from).address
-      data[:from_local]        = Mail::Address.new(from).local
-      data[:from_domain]       = Mail::Address.new(from).domain
-      data[:from_display_name] = Mail::Address.new(from).display_name ||
-                                 (Mail::Address.new(from).comments && Mail::Address.new(from).comments[0])
-    rescue
-      from.strip!
-      if from =~ /^(.+?)<(.+?)@(.+?)>$/
-        data[:from_email]        = "#{$2}@#{$3}"
-        data[:from_local]        = $2
-        data[:from_domain]       = $3
-        data[:from_display_name] = $1
-      else
-        data[:from_email]  = from
-        data[:from_local]  = from
-        data[:from_domain] = from
-      end
-    end
-
-    # do extra decoding because we needed to use field.value
-    data[:from_display_name] = Mail::Field.new('X-From', data[:from_display_name]).to_s
-    data[:from_display_name].delete!('"')
-    data[:from_display_name].strip!
-    data[:from_display_name].gsub!(/^'/, '')
-    data[:from_display_name].gsub!(/'$/, '')
+    data = data.merge(Channel::EmailParser.sender_properties(from))
 
     # compat headers
     data[:message_id] = data['message-id'.to_sym]
@@ -171,6 +156,7 @@ class Channel::EmailParser
       if data[:body].empty? && mail.text_part
         data[:body] = mail.text_part.body.decoded
         data[:body] = Encode.conv(mail.text_part.charset, data[:body])
+        data[:body] = data[:body].to_s.force_encoding('utf-8')
 
         if !data[:body].valid_encoding?
           data[:body] = data[:body].encode('utf-8', 'binary', invalid: :replace, undef: :replace, replace: '?')
@@ -323,7 +309,17 @@ class Channel::EmailParser
     # get file preferences
     headers_store = {}
     file.header.fields.each { |field|
-      headers_store[field.name.to_s] = field.value.to_s
+
+      # full line, encode, ready for storage
+      begin
+        value = Encode.conv('utf8', field.to_s)
+        if value.blank?
+          value = field.raw_value
+        end
+        headers_store[field.name.to_s] = value
+      rescue => e
+        headers_store[field.name.to_s] = field.raw_value
+      end
     }
 
     # get filename from content-disposition
@@ -334,15 +330,29 @@ class Channel::EmailParser
       filename = file.header[:content_disposition].filename
     rescue
       begin
-        result = file.header[:content_disposition].to_s.scan( /filename=("|)(.+?)("|);/i )
-        if result && result[0] && result[0][1]
-          filename = result[0][1]
+        if file.header[:content_disposition].to_s =~ /filename="(.+?)"/i
+          filename = $1
+        elsif file.header[:content_disposition].to_s =~ /filename='(.+?)'/i
+          filename = $1
+        elsif file.header[:content_disposition].to_s =~ /filename=(.+?);/i
+          filename = $1
         end
       rescue
         Rails.logger.debug 'Unable to get filename'
       end
     end
 
+    # as fallback, use raw values
+    if filename.blank?
+      if headers_store['Content-Disposition'].to_s =~ /filename="(.+?)"/i
+        filename = $1
+      elsif headers_store['Content-Disposition'].to_s =~ /filename='(.+?)'/i
+        filename = $1
+      elsif headers_store['Content-Disposition'].to_s =~ /filename=(.+?);/i
+        filename = $1
+      end
+    end
+
     # for some broken sm mail clients (X-MimeOLE: Produced By Microsoft Exchange V6.5)
     filename ||= file.header[:content_location].to_s
 
@@ -377,7 +387,8 @@ class Channel::EmailParser
 
     # cleanup content id, <> will be added automatically later
     if headers_store['Content-ID']
-      headers_store['Content-ID'].gsub!(/^</, '').gsub!(/>$/, '')
+      headers_store['Content-ID'].gsub!(/^</, '')
+      headers_store['Content-ID'].gsub!(/>$/, '')
     end
 
     # workaround for mail gem
@@ -636,6 +647,49 @@ returns
     true
   end
 
+  def self.sender_properties(from)
+    data = {}
+
+    begin
+      list = Mail::AddressList.new(from)
+      list.addresses.each { |address|
+        data[:from_email] = address.address
+        data[:from_local]        = address.local
+        data[:from_domain]       = address.domain
+        data[:from_display_name] = address.display_name ||
+                                   (address.comments && address.comments[0])
+        break if data[:from_email].present? && data[:from_email] =~ /@/
+      }
+    rescue => e
+      if from =~ /<>/ && from =~ /<.+?>/
+        data = sender_properties(from.gsub(/<>/, ''))
+      end
+    end
+
+    if data.empty? || data[:from_email].blank?
+      from.strip!
+      if from =~ /^(.+?)<(.+?)@(.+?)>$/
+        data[:from_email]        = "#{$2}@#{$3}"
+        data[:from_local]        = $2
+        data[:from_domain]       = $3
+        data[:from_display_name] = $1
+      else
+        data[:from_email]  = from
+        data[:from_local]  = from
+        data[:from_domain] = from
+      end
+    end
+
+    # do extra decoding because we needed to use field.value
+    data[:from_display_name] = Mail::Field.new('X-From', data[:from_display_name]).to_s
+    data[:from_display_name].delete!('"')
+    data[:from_display_name].strip!
+    data[:from_display_name].gsub!(/^'/, '')
+    data[:from_display_name].gsub!(/'$/, '')
+
+    data
+  end
+
   def set_attributes_by_x_headers(item_object, header_name, mail, suffix = false)
 
     # loop all x-zammad-hedaer-* headers
@@ -711,6 +765,7 @@ module Mail
   class Field
     def raw_value
       value = Encode.conv('utf8', @raw_value)
+      return value if value.blank?
       value.sub(/^.+?:(\s|)/, '')
     end
   end

app/models/channel/filter/database.rb

@@ -13,13 +13,13 @@ module Channel::Filter::Database
       min_one_rule_exists = false
       filter[:match].each { |key, meta|
         begin
-          next if !meta || !meta['value'] || meta['value'].empty?
+          next if meta.blank? || meta['value'].blank?
           min_one_rule_exists = true
-          scan = []
-          if mail
-            scan = mail[ key.downcase.to_sym ].scan(/#{meta['value']}/i)
+          has_matched = false
+          if mail[ key.downcase.to_sym ].present? && mail[ key.downcase.to_sym ] =~ /#{meta['value']}/i
+            has_matched = true
           end
-          if scan[0]
+          if has_matched
             if meta[:operator] == 'contains not'
               all_matches_ok = false
             end

app/models/channel/filter/identify_sender.rb

@@ -35,7 +35,7 @@ module Channel::Filter::IdentifySender
             items.each { |item|
 
               # skip if recipient is system email
-              next if EmailAddress.find_by(email: item.address)
+              next if EmailAddress.find_by(email: item.address.downcase)
 
               customer_user = user_create(
                 login: item.address,
@@ -93,6 +93,7 @@ module Channel::Filter::IdentifySender
         next if !mail[item.to_sym].addrs
         items = mail[item.to_sym].addrs
         items.each { |address_data|
+          next if address_data.address.blank?
           user_create(
             firstname: address_data.display_name,
             lastname: '',
@@ -102,19 +103,22 @@ module Channel::Filter::IdentifySender
       rescue => e
         # parse not parseable fields by mail gem like
         #  - Max Kohl | [example.com] <kohl@example.com>
+        #  - Max Kohl <max.kohl <max.kohl@example.com>
         Rails.logger.error 'ERROR: ' + e.inspect
-        Rails.logger.error 'ERROR: try it by my self'
+        Rails.logger.error "ERROR: try it by my self (#{item}): #{mail[item.to_sym]}"
         recipients = mail[item.to_sym].to_s.split(',')
         recipients.each { |recipient|
           address = nil
           display_name = nil
-          if recipient =~ /<(.+?)>/
+          if recipient =~ /.*<(.+?)>/
+            address = $1
+          elsif recipient =~ /<(.+?)>/
             address = $1
           end
           if recipient =~ /^(.+?)<(.+?)>/
             display_name = $1
           end
-          next if address.empty?
+          next if address.blank?
           user_create(
             firstname: display_name,
             lastname: '',
@@ -126,8 +130,9 @@ module Channel::Filter::IdentifySender
   end
 
   def self.user_create(data)
-
-    # return existing
+    if data[:email] !~ /@/
+      data[:email] += '@local'
+    end
     user = User.find_by(email: data[:email].downcase)
     if !user
       user = User.find_by(login: data[:email].downcase)
@@ -137,10 +142,7 @@ module Channel::Filter::IdentifySender
     if user
       if user.firstname.blank? && user.lastname.blank?
         if data[:firstname].present?
-          data[:firstname].strip!
-          data[:firstname].delete!('"')
-          data[:firstname].gsub!(/^'/, '')
-          data[:firstname].gsub!(/'$/, '')
+          data[:firstname] = cleanup_name(data[:firstname])
           user.update_attributes(
             firstname: data[:firstname]
           )
@@ -157,10 +159,7 @@ module Channel::Filter::IdentifySender
       if data[item.to_sym].nil?
         data[item.to_sym] = ''
       end
-      data[item.to_sym].strip!
-      data[item.to_sym].delete!('"')
-      data[item.to_sym].gsub!(/^'/, '')
-      data[item.to_sym].gsub!(/'$/, '')
+      data[item.to_sym] = cleanup_name(data[item.to_sym])
     }
     data[:password]      = ''
     data[:active]        = true
@@ -176,4 +175,13 @@ module Channel::Filter::IdentifySender
     user
   end
 
+  def self.cleanup_name(string)
+    string.strip!
+    string.delete!('"')
+    string.gsub!(/^'/, '')
+    string.gsub!(/'$/, '')
+    string.gsub!(/.+?\s\(.+?\)$/, '')
+    string
+  end
+
 end

app/models/channel/filter/sender_is_system_address.rb

@@ -5,13 +5,13 @@ module Channel::Filter::SenderIsSystemAddress
   def self.run(_channel, mail)
 
     # if attributes already set by header
-    return if mail[ 'x-zammad-ticket-create-article-sender'.to_sym ]
-    return if mail[ 'x-zammad-article-sender'.to_sym ]
+    return if mail['x-zammad-ticket-create-article-sender'.to_sym]
+    return if mail['x-zammad-article-sender'.to_sym]
 
     # check if sender address is system
     form = 'raw-from'.to_sym
-    return if !mail[form]
-    return if !mail[:to]
+    return if mail[form].blank?
+    return if mail[:to].blank?
 
     # in case, set sender
     begin
@@ -19,8 +19,8 @@ module Channel::Filter::SenderIsSystemAddress
       items = mail[form].addrs
       items.each { |item|
         next if !EmailAddress.find_by(email: item.address.downcase)
-        mail[ 'x-zammad-ticket-create-article-sender'.to_sym ] = 'Agent'
-        mail[ 'x-zammad-article-sender'.to_sym ] = 'Agent'
+        mail['x-zammad-ticket-create-article-sender'.to_sym] = 'Agent'
+        mail['x-zammad-article-sender'.to_sym] = 'Agent'
         return true
       }
     rescue => e
@@ -28,13 +28,13 @@ module Channel::Filter::SenderIsSystemAddress
     end
 
     # check if sender is agent
-    return if mail[:from_email].empty?
+    return if mail[:from_email].blank?
     begin
-      user = User.find_by(email: mail[:from_email])
+      user = User.find_by(email: mail[:from_email].downcase)
       return if !user
       return if !user.permissions?('ticket.agent')
-      mail[ 'x-zammad-ticket-create-article-sender'.to_sym ] = 'Agent'
-      mail[ 'x-zammad-article-sender'.to_sym ] = 'Agent'
+      mail['x-zammad-ticket-create-article-sender'.to_sym] = 'Agent'
+      mail['x-zammad-article-sender'.to_sym] = 'Agent'
       return true
     rescue => e
       Rails.logger.error 'ERROR: SenderIsSystemAddress: ' + e.inspect

app/models/concerns/search_indexed.rb

@@ -123,7 +123,9 @@ reload search index with full data
     def search_index_reload
       tolerance       = 5
       tolerance_count = 0
-      all.order('created_at DESC').each { |item|
+      ids = all.order('created_at DESC').pluck(:id)
+      ids.each { |item_id|
+        item = find(item_id)
         next if item.ignore_search_indexing?(:destroy)
         begin
           item.search_index_update_backend

app/models/cti/caller_id.rb

@@ -54,8 +54,10 @@ returns
           search_params[:level] = level
         end
 
-        result = Cti::CallerId.where(search_params).group(:user_id, :id).order(id: 'DESC').limit(20)
-
+        caller_ids = Cti::CallerId.select('MAX(id) as caller_id').where(search_params).group(:user_id).order('caller_id DESC').limit(20).map(&:caller_id)
+        Cti::CallerId.where(id: caller_ids).order(id: :desc).each { |record|
+          result.push record
+        }
         break if result.present?
       }
       result

app/models/cti/log.rb

@@ -266,7 +266,22 @@ returns
       }
     end
 
-    # processes a incoming event
+=begin
+
+processes a incoming event
+
+Cti::Log.process(
+  'cause' => '',
+  'event' => 'newCall',
+  'user' => 'user 1',
+  'from' => '4912347114711',
+  'to' => '4930600000000',
+  'callId' => '4991155921769858278-1',
+  'direction' => 'in',
+)
+
+=end
+
     def self.process(params)
       comment = params['cause']
       event   = params['event']

app/models/email_address.rb

@@ -8,6 +8,7 @@ class EmailAddress < ApplicationModel
   validates       :realname, presence: true
   validates       :email,    presence: true
 
+  before_validation :check_email
   before_create   :check_if_channel_exists_set_inactive
   before_update   :check_if_channel_exists_set_inactive
   after_create    :update_email_address_id
@@ -41,6 +42,15 @@ check and if channel not exists reset configured channels for email addresses
 
   private
 
+  def check_email
+    return true if Setting.get('import_mode')
+    return true if email.blank?
+    self.email = email.downcase.strip
+    raise Exceptions::UnprocessableEntity, 'Invalid email' if email !~ /@/
+    raise Exceptions::UnprocessableEntity, 'Invalid email' if email =~ /\s/
+    true
+  end
+
   # set email address to inactive/active if channel exists or not
   def check_if_channel_exists_set_inactive
 

app/models/import_job.rb

@@ -18,7 +18,8 @@ class ImportJob < ApplicationModel
   def start
     self.started_at = Time.zone.now
     save
-    name.constantize.new(self)
+    instance = name.constantize.new(self)
+    instance.start
   rescue => e
     Rails.logger.error e
 
@@ -33,6 +34,25 @@ class ImportJob < ApplicationModel
     save
   end
 
+  # Gets called when the Scheduler gets (re-)started and this job was still
+  # in the queue. If `finished_at` is blank the call is piped through to
+  # the ImportJob backend which has to decide how to go from here. The delayed
+  # job will get destroyed if rescheduled? is not implemented
+  # as an ImportJob backend class method.
+  #
+  # @see Scheduler#cleanup_delayed
+  #
+  # @example
+  #  import.reschedule?(delayed_job)
+  #
+  # return [Boolean] whether the ImportJob should get rescheduled (true) or destroyed (false)
+  def reschedule?(delayed_job)
+    return false if finished_at.present?
+    instance = name.constantize.new(self)
+    return false if !instance.respond_to?(:reschedule?)
+    instance.reschedule?(delayed_job)
+  end
+
   # Convenience wrapper around the start method for starting (delayed) dry runs.
   # Logs the start and end time (if ended successfully) and logs
   # exceptions into result if they happen.
@@ -81,7 +101,7 @@ class ImportJob < ApplicationModel
   end
 
   # Queues all configured import backends from Setting 'import_backends' as import jobs
-  # that are not yet queued.
+  # that are not yet queued. Backends which are not #queueable? are skipped.
   #
   # @example
   #  ImportJob.queue_registered
@@ -98,6 +118,9 @@ class ImportJob < ApplicationModel
         next
       end
 
+      # skip backends that are not "ready" yet
+      next if !backend.constantize.queueable?
+
       # skip if no entry exists
       # skip if a not finished entry exists
       next if ImportJob.exists?(name: backend, finished_at: nil)

app/models/object_manager/attribute.rb

@@ -227,6 +227,11 @@ possible types
 
       # if data_option has changed, store it for next migration
       if !force
+        [:name, :display, :data_type, :position, :active].each { |key|
+          next if record[key] == data[key]
+          data[:to_config] = true
+          break
+        }
         if record[:data_option] != data[:data_option]
 
           # do we need a database migration?
@@ -587,7 +592,9 @@ to send no browser reload event, pass false
         end
 
         # restart processes
+        attribute.to_create = false
         attribute.to_migrate = false
+        attribute.to_delete = false
         attribute.save!
         reset_database_info(model)
         execute_db_count += 1
@@ -661,6 +668,9 @@ to send no browser reload event, pass false
   def self.reset_database_info(model)
     model.connection.schema_cache.clear!
     model.reset_column_information
+    # rebuild columns cache to reduce the risk of
+    # race conditions in re-setting it with outdated data
+    model.columns
   end
 
   def check_name

app/models/observer/ticket/article_changes.rb

@@ -41,7 +41,7 @@ class Observer::Ticket::ArticleChanges < ActiveRecord::Observer
       record.ticket.touch
       return
     end
-    record.ticket.save
+    record.ticket.save!
   end
 
   # get article count

app/models/observer/ticket/last_owner_update.rb

@@ -0,0 +1,34 @@
+# Copyright (C) 2012-2016 Zammad Foundation, http://zammad-foundation.org/
+
+class Observer::Ticket::LastOwnerUpdate < ActiveRecord::Observer
+  observe 'ticket'
+
+  def before_create(record)
+    _check('create', record)
+  end
+
+  def before_update(record)
+    _check('update', record)
+  end
+
+  private
+
+  def _check(type, record)
+
+    # return if we run import mode
+    return true if Setting.get('import_mode')
+
+    # check if owner has changed
+    if type == 'update'
+      return true if record.changes['owner_id'].blank?
+    end
+
+    # check if owner is nobody
+    if record.owner_id.blank? || record.owner_id == 1
+      record.last_owner_update_at = nil
+      return true
+    end
+
+    record.last_owner_update_at = Time.zone.now
+  end
+end

app/models/overview.rb

@@ -25,22 +25,35 @@ class Overview < ApplicationModel
   end
 
   def fill_link_on_create
-    return true if !link.empty?
+    return true if link.present?
     self.link = link_name(name)
   end
 
   def fill_link_on_update
-    return true if link.empty?
     return true if !changes['name']
+    return true if changes['link']
     self.link = link_name(name)
   end
 
   def link_name(name)
-    link = name.downcase
-    link.gsub!(/\s/, '_')
-    link.gsub!(/[^0-9a-z]/i, '_')
-    link.gsub!(/_+/, '_')
-    link
+    local_link = name.downcase
+    local_link = local_link.parameterize('_')
+    local_link.gsub!(/\s/, '_')
+    local_link.gsub!(/_+/, '_')
+    local_link = URI.escape(local_link)
+    if local_link.blank?
+      local_link = id || rand(999)
+    end
+    check = true
+    while check
+      exists = Overview.find_by(link: local_link)
+      if exists && exists.id != id
+        local_link = "#{local_link}_#{rand(999)}"
+      else
+        check = false
+      end
+    end
+    local_link
   end
 
 end

app/models/scheduler.rb

@@ -11,6 +11,18 @@ class Scheduler < ApplicationModel
 
     Thread.abort_on_exception = true
 
+    # reconnect in case db connection is lost
+    # See issue #1080
+    begin
+      ActiveRecord::Base.connection.reconnect!
+    rescue PG::UnableToSend => e # rubocop:disable Lint/HandleExceptions
+    rescue => e
+      logger.error "Can't reconnect to database #{e.inspect}"
+    end
+
+    # cleanup old background jobs
+    cleanup
+
     # start worker for background jobs
     worker
 
@@ -44,6 +56,80 @@ class Scheduler < ApplicationModel
     end
   end
 
+  # Checks all delayed jobs that are locked and cleans them up.
+  # Should only get called when the Scheduler gets started.
+  #
+  # @see Scheduler#cleanup_delayed
+  #
+  # @param [Boolean] force forces the cleanup if not called in Scheduler starting context.
+  #
+  # @example
+  #   Scheduler.cleanup
+  #
+  # @raise [RuntimeError] If called without force and not when Scheduler gets started.
+  #
+  # return [nil]
+  def self.cleanup(force: false)
+
+    if !force && caller_locations.first.label != 'threads'
+      raise 'This method should only get called when Scheduler.threads are initialized. Use `force: true` to start anyway.'
+    end
+
+    Delayed::Job.all.each do |job|
+      cleanup_delayed(job)
+    end
+  end
+
+  # Checks if the given job can be rescheduled or destroys it. Logs the action as warn.
+  # Works only for locked jobs. Jobs that are not locked are ignored and
+  # should get destroyed directly.
+  # Checks the delayed job object for a method called .reschedule?. The memthod is called
+  # with the delayed job as a parameter. The result value is expected as a Boolean. If the
+  # result is true the lock gets removed and the delayed job gets rescheduled. If the return
+  # value is false it will get destroyed which is the default behaviour.
+  #
+  # @param [Delayed::Job] job the job that should get checked for destroying/rescheduling.
+  #
+  # @example
+  #   Scheduler.cleanup_delayed(job)
+  #
+  # return [nil]
+  def self.cleanup_delayed(job)
+    return if job.locked_at.blank?
+
+    job_name       = job.name
+    payload_object = job.payload_object
+    reschedule     = false
+    if payload_object.present?
+      if payload_object.respond_to?(:object)
+        object = payload_object.object
+
+        if object.respond_to?(:id)
+          job_name += " (id: #{object.id})"
+        end
+
+        if object.respond_to?(:reschedule?) && object.reschedule?(job)
+          reschedule = true
+        end
+      end
+
+      if payload_object.respond_to?(:args)
+        job_name += " - ARGS: #{payload_object.args.inspect}"
+      end
+    end
+
+    if reschedule
+      action = 'Rescheduling'
+      job.unlock
+      job.save
+    else
+      action = 'Destroyed'
+      job.destroy
+    end
+
+    Rails.logger.warn "#{action} locked delayed job: #{job_name}"
+  end
+
   def self.start_job(job)
 
     Thread.new {

app/models/signature.rb

@@ -2,7 +2,11 @@
 
 class Signature < ApplicationModel
   include LatestChangeObserved
+  include HtmlSanitized
 
   has_many  :groups,  after_add: :cache_update, after_remove: :cache_update
   validates :name,    presence: true
+
+  sanitized_html :body
+
 end

app/models/store.rb

@@ -139,6 +139,19 @@ returns
     true
   end
 
+=begin
+
+get content of file
+
+  store = Store.find(store_id)
+  content_as_string = store.content
+
+returns
+
+  content_as_string
+
+=end
+
   def content
     file = Store::File.find_by(id: store_file_id)
     if !file
@@ -147,6 +160,34 @@ returns
     file.content
   end
 
+=begin
+
+get content of file
+
+  store = Store.find(store_id)
+  location_of_file = store.save_to_file
+
+returns
+
+  location_of_file
+
+=end
+
+  def save_to_file(path = nil)
+    content
+    file = Store::File.find_by(id: store_file_id)
+    if !file
+      raise "No such file #{store_file_id}!"
+    end
+    if !path
+      path = "#{Rails.root}/tmp/#{filename}"
+    end
+    ::File.open(path, 'wb') { |handle|
+      handle.write file.content
+    }
+    path
+  end
+
   def provider
     file = Store::File.find_by(id: store_file_id)
     if !file

app/models/tag.rb

@@ -211,10 +211,13 @@ rename tag items
 
     def self.rename(data)
 
-      new_tag_name = data[:name].strip
-      old_tag_item = Tag::Item.find(data[:id])
+      new_tag_name         = data[:name].strip
+      old_tag_item         = Tag::Item.find(data[:id])
       already_existing_tag = Tag::Item.lookup(name: new_tag_name)
 
+      # check if no remame is needed
+      return true if new_tag_name == old_tag_item.name
+
       # merge old with new tag if already existing
       if already_existing_tag
 
@@ -245,22 +248,21 @@ rename tag items
 
         # delete not longer used tag
         old_tag_item.destroy
+        return true
+      end
 
       # update new tag name
-      else
+      old_tag_item.name = new_tag_name
+      old_tag_item.save
 
-        old_tag_item.name = new_tag_name
-        old_tag_item.save
-
-        # touch reference objects
-        Tag.where(tag_item_id: old_tag_item.id).each { |tag|
-          tag_object = Tag::Object.lookup(id: tag.tag_object_id)
-          Tag.touch_reference_by_params(
-            object: tag_object.name,
-            o_id: tag.o_id,
-          )
-        }
-      end
+      # touch reference objects
+      Tag.where(tag_item_id: old_tag_item.id).each { |tag|
+        tag_object = Tag::Object.lookup(id: tag.tag_object_id)
+        Tag.touch_reference_by_params(
+          object: tag_object.name,
+          o_id: tag.o_id,
+        )
+      }
 
       true
     end

app/models/text_module.rb

@@ -2,10 +2,13 @@
 
 class TextModule < ApplicationModel
   include NotifiesClients
+  include HtmlSanitized
 
   validates :name,    presence: true
   validates :content, presence: true
 
+  sanitized_html :content
+
 =begin
 
 load text modules from online

app/models/ticket.rb

@@ -47,6 +47,7 @@ class Ticket < ApplicationModel
                                      :last_contact_at,
                                      :last_contact_agent_at,
                                      :last_contact_customer_at,
+                                     :last_owner_update_at,
                                      :preferences
 
   history_attributes_ignored :create_article_type_id,
@@ -144,7 +145,7 @@ returns
     pending_action = Ticket::StateType.find_by(name: 'pending action')
     ticket_states_pending_action = Ticket::State.where(state_type_id: pending_action)
                                                 .where.not(next_state_id: nil)
-    if !ticket_states_pending_action.empty?
+    if ticket_states_pending_action.present?
       next_state_map = {}
       ticket_states_pending_action.each { |state|
         next_state_map[state.id] = state.next_state_id
@@ -168,7 +169,7 @@ returns
     pending_reminder = Ticket::StateType.find_by(name: 'pending reminder')
     ticket_states_pending_reminder = Ticket::State.where(state_type_id: pending_reminder)
 
-    if !ticket_states_pending_reminder.empty?
+    if ticket_states_pending_reminder.present?
       reminder_state_map = {}
       ticket_states_pending_reminder.each { |state|
         reminder_state_map[state.id] = state.next_state_id
@@ -259,6 +260,47 @@ returns
 
 =begin
 
+processes tickets which auto unassign time has reached
+
+  processed_tickets = Ticket.process_auto_unassign
+
+returns
+
+  processed_tickets = [<Ticket>, ...]
+
+=end
+
+  def self.process_auto_unassign
+
+    # process pending action tickets
+    state_ids = Ticket::State.by_category(:work_on).pluck(:id)
+    return [] if state_ids.blank?
+    result = []
+    groups = Group.where(active: true).where('assignment_timeout IS NOT NULL AND groups.assignment_timeout != 0')
+    return [] if groups.blank?
+    groups.each { |group|
+      next if group.assignment_timeout.blank?
+      ticket_ids = Ticket.where('state_id IN (?) AND owner_id != 1 AND group_id = ?', state_ids, group.id).limit(600).pluck(:id)
+      ticket_ids.each { |ticket_id|
+        ticket = Ticket.find_by(id: ticket_id)
+        next if !ticket
+        minutes_since_last_assignment = Time.zone.now - ticket.last_owner_update_at
+        next if (minutes_since_last_assignment / 60) <= group.assignment_timeout
+        Transaction.execute do
+          ticket.owner_id      = 1
+          ticket.updated_at    = Time.zone.now
+          ticket.updated_by_id = 1
+          ticket.save!
+        end
+        result.push ticket
+      }
+    }
+
+    result
+  end
+
+=begin
+
 merge tickets
 
   ticket = Ticket.find(123)
@@ -275,6 +317,14 @@ returns
 
   def merge_to(data)
 
+    # prevent cross merging tickets
+    target_ticket = Ticket.find_by(id: data[:ticket_id])
+    raise 'no target ticket given' if !target_ticket
+    raise Exceptions::UnprocessableEntity, 'ticket already merged, no merge into merged ticket possible' if target_ticket.state.state_type.name == 'merged'
+
+    # check different ticket ids
+    raise Exceptions::UnprocessableEntity, 'Can\'t merge ticket with it self!' if id == target_ticket.id
+
     # update articles
     Transaction.execute do
 
@@ -317,7 +367,7 @@ returns
       save!
 
       # touch new ticket (to broadcast change)
-      Ticket.find(data[:ticket_id]).touch
+      target_ticket.touch
     end
     true
   end
@@ -390,17 +440,26 @@ get count of tickets and tickets which match on selector
     query, bind_params, tables = selector2sql(selectors, current_user)
     return [] if !query
 
-    if !current_user
-      ticket_count = Ticket.where(query, *bind_params).joins(tables).count
-      tickets = Ticket.where(query, *bind_params).joins(tables).limit(limit)
-      return [ticket_count, tickets]
+    ActiveRecord::Base.transaction(requires_new: true) do
+      begin
+        if !current_user
+          ticket_count = Ticket.where(query, *bind_params).joins(tables).count
+          tickets = Ticket.where(query, *bind_params).joins(tables).limit(limit)
+          return [ticket_count, tickets]
+        end
+
+        access_condition = Ticket.access_condition(current_user)
+        ticket_count = Ticket.where(access_condition).where(query, *bind_params).joins(tables).count
+        tickets = Ticket.where(access_condition).where(query, *bind_params).joins(tables).limit(limit)
+
+        return [ticket_count, tickets]
+      rescue ActiveRecord::StatementInvalid => e
+        Rails.logger.error e.inspect
+        Rails.logger.error e.backtrace
+        raise ActiveRecord::Rollback
+      end
     end
-
-    access_condition = Ticket.access_condition(current_user)
-    ticket_count = Ticket.where(access_condition).where(query, *bind_params).joins(tables).count
-    tickets = Ticket.where(access_condition).where(query, *bind_params).joins(tables).limit(limit)
-
-    [ticket_count, tickets]
+    []
   end
 
 =begin
@@ -778,10 +837,48 @@ perform changes on ticket
           if item && item[:article_id]
             article = Ticket::Article.lookup(id: item[:article_id])
             if article && article.preferences['is-auto-response'] == true && article.from && article.from =~ /#{Regexp.quote(user.email)}/i
-              logger.info "Send not trigger based notification to #{user.email} because of auto response tagged incoming email"
+              logger.info "Send no trigger based notification to #{user.email} because of auto response tagged incoming email"
               next
             end
           end
+          recipient_email = user.email.downcase.strip
+
+          # loop protection / check if maximal count of trigger mail has reached
+          map = {
+            30 => 15,
+            60 => 25,
+            180 => 50,
+          }
+          skip = false
+          map.each { |minutes, count|
+            already_sent = Ticket::Article.where(
+              ticket_id: id,
+              sender: Ticket::Article::Sender.find_by(name: 'System'),
+              type: Ticket::Article::Type.find_by(name: 'email'),
+            ).where("ticket_articles.created_at > ? AND ticket_articles.to LIKE '%#{recipient_email.strip}%'", Time.zone.now - minutes.minutes).count
+            next if already_sent < count
+            logger.info "Send no trigger based notification to #{recipient_email} because already sent #{count} for this ticket within last #{minutes} minutes (loop protection)"
+            skip = true
+            break
+          }
+          next if skip
+          map = {
+            1 => 150,
+            3 => 250,
+            6 => 450,
+          }
+          skip = false
+          map.each { |hours, count|
+            already_sent = Ticket::Article.where(
+              sender: Ticket::Article::Sender.find_by(name: 'System'),
+              type: Ticket::Article::Type.find_by(name: 'email'),
+            ).where("ticket_articles.created_at > ? AND ticket_articles.to LIKE '%#{recipient_email.strip}%'", Time.zone.now - hours.hours).count
+            next if already_sent < count
+            logger.info "Send no trigger based notification to #{recipient_email} because already sent #{count} in total within last #{hours} hour(s) (loop protection)"
+            skip = true
+            break
+          }
+          next if skip
 
           email = user.email.downcase.strip
           next if recipient_already[email]
@@ -989,6 +1086,10 @@ result
     # ignore if no state has changed
     return if !changes['state_id']
 
+    # ignore if new state is blank and
+    # let handle ActiveRecord the error
+    return if state_id.blank?
+
     # check if new state isn't pending*
     current_state      = Ticket::State.lookup(id: state_id)
     current_state_type = Ticket::StateType.lookup(id: current_state.state_type_id)

app/models/ticket/article.rb

@@ -15,8 +15,8 @@ class Ticket::Article < ApplicationModel
   belongs_to    :created_by,  class_name: 'User'
   belongs_to    :updated_by,  class_name: 'User'
   store         :preferences
-  before_create :check_subject, :check_message_id_md5
-  before_update :check_subject, :check_message_id_md5
+  before_create :check_subject, :check_body, :check_message_id_md5
+  before_update :check_subject, :check_body, :check_message_id_md5
 
   sanitized_html :body
 
@@ -285,6 +285,22 @@ returns
     subject.gsub!(/\s|\t|\r/, ' ')
   end
 
+  # strip body length or raise exception
+  def check_body
+    return true if body.blank?
+    limit = 1_500_000
+    current_length = body.length
+    if body.length > limit
+      if ApplicationHandleInfo.current.present? && ApplicationHandleInfo.current.split('.')[1] == 'postmaster'
+        logger.warn "WARNING: cut string because of database length #{self.class}.body(#{limit} but is #{current_length})"
+        self.body = body[0, limit]
+      else
+        raise Exceptions::UnprocessableEntity, "body if article is to large, #{current_length} chars - only #{limit} allowed"
+      end
+    end
+    true
+  end
+
   def history_log_attributes
     {
       related_o_id:           self['ticket_id'],

app/models/ticket/overviews.rb

@@ -5,9 +5,7 @@ module Ticket::Overviews
 
 all overviews by user
 
-  result = Ticket::Overviews.all(
-    current_user: User.find(123),
-  )
+  result = Ticket::Overviews.all(current_user: User.find(123))
 
 returns
 

app/models/ticket/subject.rb

@@ -22,22 +22,28 @@ returns
     ticket_hook         = Setting.get('ticket_hook')
     ticket_hook_divider = Setting.get('ticket_hook_divider')
     ticket_subject_re   = Setting.get('ticket_subject_re')
-    if is_reply && !ticket_subject_re.empty?
-      subject = "#{ticket_subject_re}: #{subject}"
-    end
 
     # none position
     if Setting.get('ticket_hook_position') == 'none'
+      if is_reply && ticket_subject_re.present?
+        subject = "#{ticket_subject_re}: #{subject}"
+      end
       return subject
     end
 
     # right position
     if Setting.get('ticket_hook_position') == 'right'
-      return subject + " [#{ticket_hook}#{ticket_hook_divider}#{number}]"
+      if is_reply && ticket_subject_re.present?
+        subject = "#{ticket_subject_re}: #{subject}"
+      end
+      return "#{subject} [#{ticket_hook}#{ticket_hook_divider}#{number}]"
     end
 
     # left position
-    "[#{ticket_hook}#{ticket_hook_divider}#{number}] " + subject
+    if is_reply && ticket_subject_re.present?
+      return "#{ticket_subject_re}: [#{ticket_hook}#{ticket_hook_divider}#{number}] #{subject}"
+    end
+    "[#{ticket_hook}#{ticket_hook_divider}#{number}] #{subject}"
   end
 
 =begin

app/models/transaction/karma.rb

@@ -54,7 +54,7 @@ class Transaction::Karma
 
     if @item[:type] == 'reminder_reached'
       return if ticket.owner_id == 1
-      return if ticket.pending_time > Time.zone.now - 2.days
+      return if ticket.pending_time && ticket.pending_time > Time.zone.now - 2.days
       Karma::ActivityLog.add('ticket reminder overdue (+2 days)', ticket.owner, 'Ticket', ticket.id)
       return
     end

app/models/transaction/trigger.rb

@@ -130,6 +130,7 @@ class Transaction::Trigger
 
         # verify is condition is matching
         ticket_count, tickets = Ticket.selectors(condition, 1)
+        next if ticket_count.blank?
         next if ticket_count.zero?
         next if tickets.first.id != ticket.id
         user_id = ticket.updated_by_id

app/models/user.rb

@@ -42,7 +42,7 @@ class User < ApplicationModel
   before_update   :check_preferences_default, :validate_roles, :reset_login_failed
   after_create    :avatar_for_email_check
   after_update    :avatar_for_email_check
-  after_destroy   :avatar_destroy
+  after_destroy   :avatar_destroy, :user_device_destroy
 
   has_and_belongs_to_many :groups,          after_add: :cache_update, after_remove: :cache_update, class_name: 'Group'
   has_and_belongs_to_many :roles,           after_add: [:cache_update, :check_notifications], after_remove: :cache_update, before_add: :validate_agent_limit, before_remove: :last_admin_check, class_name: 'Role'
@@ -903,6 +903,8 @@ raise 'Minimum one user need to have admin permissions'
 =end
 
   def last_admin_check(role)
+    return if Setting.get('import_mode')
+
     ticket_admin_role_ids = Role.joins(:permissions).where(permissions: { name: ['admin', 'admin.user'] }).pluck(:id)
     count                 = User.joins(:roles).where(roles: { id: ticket_admin_role_ids }, users: { active: true }).count
     if ticket_admin_role_ids.include?(role.id)
@@ -976,6 +978,10 @@ raise 'Minimum one user need to have admin permissions'
     Avatar.remove('User', id)
   end
 
+  def user_device_destroy
+    UserDevice.remove(id)
+  end
+
   def ensure_password
     return if password_empty?
     return if PasswordHash.crypted?(password)

app/models/user_device.rb

@@ -206,4 +206,15 @@ send user notification about new device or new location for device
     )
   end
 
+=begin
+
+delete device devices of user
+
+  user_devices = UserDevice.remove(user.id)
+
+=end
+
+  def self.remove(user_id)
+    UserDevice.where(user_id: user_id).destroy_all
+  end
 end

config/application.rb

@@ -23,6 +23,7 @@ module Zammad
     config.active_record.observers =
       'observer::_session',
       'observer::_ticket::_close_time',
+      'observer::_ticket::_last_owner_update',
       'observer::_ticket::_user_ticket_counter',
       'observer::_ticket::_article_changes',
       'observer::_ticket::_article::_fillup_from_general',

config/initializers/db_preferences_mysql.rb

@@ -1,6 +1,7 @@
 # mysql
 if ActiveRecord::Base.connection_config[:adapter] == 'mysql2'
   Rails.application.config.db_4bytes_utf8 = false
+  Rails.application.config.db_null_byte = true
 
   # mysql version check
   #  mysql example: "5.7.3"

config/initializers/db_preferences_postgresql.rb

@@ -2,6 +2,7 @@
 if ActiveRecord::Base.connection_config[:adapter] == 'postgresql'
   Rails.application.config.db_case_sensitive = true
   Rails.application.config.db_like = 'ILIKE'
+  Rails.application.config.db_null_byte = false
 
   # postgresql version check
   #  example output: "9.5.0"

config/initializers/filter_parameter_logging.rb

@@ -1,4 +1,4 @@
 # Be sure to restart your server when you modify this file.
 
 # Configure sensitive parameters which will be filtered from the log file.
-Rails.application.config.filter_parameters += [:password]
+Rails.application.config.filter_parameters += [:password, :bind_pw]

config/initializers/html_email_style.rb

@@ -8,9 +8,6 @@ Rails.application.config.html_email_body = <<HERE
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
     <style type="text/css">
       body {
-        width: 90% !important;
-        -webkit-text-size-adjust: 90%;
-        -ms-text-size-adjust: 90%;
         #{Rails.application.config.html_email_css_font};
       }
       img {
@@ -35,9 +32,6 @@ Rails.application.config.html_email_body = <<HERE
         overflow: auto;
         word-break: keep-all;
       }
-      p, table, div, td {
-        max-width: 600px;
-      }
       table,
       pre,
       blockquote {
@@ -80,7 +74,7 @@ Rails.application.config.html_email_body = <<HERE
         overflow: auto;
       }
     </style>
-  <head>
+  </head>
   <body style="#{Rails.application.config.html_email_css_font}">###html###</body>
 </html>
 HERE

config/initializers/omniauth.rb

@@ -32,7 +32,6 @@ Rails.application.config.middleware.use OmniAuth::Builder do
       authorize_url: '/oauth/authorize',
       token_url: '/oauth/token'
     },
-    scope: 'read_user',
   }
 
   # oauth2 database connect

config/routes/form.rb

@@ -2,7 +2,7 @@ Zammad::Application.routes.draw do
   api_path = Rails.configuration.api_path
 
   # forms
-  match api_path + '/form_submit',      to: 'form#submit',    via: :post
-  match api_path + '/form_config',      to: 'form#config',    via: :get
+  match api_path + '/form_submit',      to: 'form#submit',        via: :post
+  match api_path + '/form_config',      to: 'form#configuration', via: :get
 
 end

contrib/backup/config

@@ -1,10 +0,0 @@
-#!/bin/bash
-#
-# zammad backup script config
-#
-
-ZAMMAD_DIR="/opt/zammad"
-DATABASE_CONFIG="config/database.yml"
-BACKUP_DIR="/var/tmp/zammad_backup"
-HOLD_DAYS="10"
-DEBUG="no"

contrib/backup/config.dist

@@ -0,0 +1,8 @@
+#!/bin/bash
+#
+# zammad backup script config
+#
+
+BACKUP_DIR='/var/tmp/zammad_backup'
+HOLD_DAYS='10'
+DEBUG='no'

contrib/backup/functions

@@ -3,6 +3,10 @@
 # zammad backup script functions
 #
 
+function get_zammad_dir () {
+  ZAMMAD_DIR="$(echo ${BACKUP_SCRIPT_PATH} | sed -e 's#/contrib/backup.*##g')"
+}
+
 function get_backup_date () {
   TIMESTAMP="$(date +'%Y%m%d%H%M%S')"
 
@@ -16,10 +20,10 @@ function delete_old_backups () {
 }
 
 function get_db_credentials () {
-  DB_ADAPTER="$(grep 'adapter:' < ${ZAMMAD_DIR}/${DATABASE_CONFIG} | sed -e 's/.*adapter:\(\| \)//g')"
-  DB_NAME="$(grep 'database:' < ${ZAMMAD_DIR}/${DATABASE_CONFIG} | sed -e 's/.*database:\(\| \) //g')"
-  DB_USER="$(grep 'username:' < ${ZAMMAD_DIR}/${DATABASE_CONFIG} | sed -e 's/.*username:\(\| \)//g')"
-  DB_PASS="$(grep 'password:' < ${ZAMMAD_DIR}/${DATABASE_CONFIG} | sed -e 's/.*password:\(\| \)//g')"
+  DB_ADAPTER="$(grep 'adapter:' < ${ZAMMAD_DIR}/config/database.yml | sed -e 's/.*adapter:\(\| \)//g')"
+  DB_NAME="$(grep 'database:' < ${ZAMMAD_DIR}/config/database.yml | sed -e 's/.*database:\(\| \) //g')"
+  DB_USER="$(grep 'username:' < ${ZAMMAD_DIR}/config/database.yml | sed -e 's/.*username:\(\| \)//g')"
+  DB_PASS="$(grep 'password:' < ${ZAMMAD_DIR}/config/database.yml | sed -e 's/.*password:\(\| \)//g')"
 
   if [ "${DEBUG}" == "yes" ]; then
     echo "adapter=${DB_ADAPTER} dbname=${DB_NAME} dbuser=${DB_USER} dbpass=${DB_PASS}"
@@ -35,25 +39,28 @@ function backup_dir_create () {
 }
 
 function backup_files () {
+  echo "creating file backup..."
   tar -czf ${BACKUP_DIR}/${TIMESTAMP}_zammad_files.tar.gz ${ZAMMAD_DIR}
 }
 
 function backup_db () {
   if [ "${DB_ADAPTER}" == "mysql2" ]; then
+    echo "creating mysql backup..."
     mysqldump --opt --single-transaction -u${DB_USER} -p${DB_PASS} ${DB_NAME} | gzip > ${BACKUP_DIR}/${TIMESTAMP}_zammad_db.mysql.gz
   elif [ "${DB_ADAPTER}" == "postgresql" ]; then
+    echo "creating postgresql backup..."
     su -c "pg_dump -c ${DB_NAME} | gzip > /tmp/${TIMESTAMP}_zammad_db.psql.gz" postgres
     mv /tmp/${TIMESTAMP}_zammad_db.psql.gz ${BACKUP_DIR}
   else
-    echo "DB ADAPTER not found. if its sqlite backup is already saved in filebackup"
+    echo "DB ADAPTER not found. if its sqlite backup is already saved in the filebackup"
   fi
 }
 
 function check_database_config_exists () {
-  if [ -f ${ZAMMAD_DIR}/${DATABASE_CONFIG} ]; then
+  if [ -f ${ZAMMAD_DIR}/config/database.yml ]; then
     get_db_credentials
   else
-    echo -e "${ZAMMAD_DIR}/${DATABASE_CONFIG} is missing. is zammad configured yet? \nAborting restore..."
+    echo -e "${ZAMMAD_DIR}/config/database.yml is missing. is zammad configured yet? \nAborting..."
     exit 1
   fi
 }
@@ -155,6 +162,18 @@ function restore_zammad () {
   fi
 }
 
-function restore_message () {
-    echo "# Zammad restored!"
+function start_backup_message () {
+    echo -e "\n# Zammad backup started - $(date)!\n"
+}
+
+function start_restore_message () {
+    echo -e "\n# Zammad restored started - $(date)!\n"
+}
+
+function finished_backup_message () {
+    echo -e "\n# Zammad backuped successfully - $(date)!\n"
+}
+
+function finished_restore_message () {
+    echo -e "\n# Zammad restored successfully - $(date)!\n"
 }

contrib/backup/zammad_backup.sh

@@ -4,14 +4,25 @@
 #
 
 PATH=/sbin:/bin:/usr/sbin:/usr/bin:
+BACKUP_SCRIPT_PATH="$(dirname $(realpath $0))"
 
-# import config
-. /opt/zammad/contrib/backup/config
+if [ -f "${BACKUP_SCRIPT_PATH}/config" ]; then
+  # import config
+  . ${BACKUP_SCRIPT_PATH}/config
+else
+  echo -e "\n The 'config' file is missing!"
+  echo -e " Please copy ${BACKUP_SCRIPT_PATH}/config.dist to  ${BACKUP_SCRIPT_PATH}/config before running $0!\n"
+  exit 1
+fi
 
 # import functions
-. /opt/zammad/contrib/backup/functions
+. ${BACKUP_SCRIPT_PATH}/functions
 
 # exec backup
+start_backup_message
+
+get_zammad_dir
+
 check_database_config_exists
 
 delete_old_backups
@@ -23,3 +34,5 @@ backup_dir_create
 backup_files
 
 backup_db
+
+finished_backup_message

contrib/backup/zammad_restore.sh

@@ -4,14 +4,25 @@
 #
 
 PATH=/sbin:/bin:/usr/sbin:/usr/bin:
+BACKUP_SCRIPT_PATH="$(dirname $(realpath $0))"
 
-# import config
-. /opt/zammad/contrib/backup/config
+if [ -f "${BACKUP_SCRIPT_PATH}/config" ]; then
+  # import config
+  . ${BACKUP_SCRIPT_PATH}/config
+else
+  echo -e "\n The 'config' file is missing!"
+  echo -e " Please copy ${BACKUP_SCRIPT_PATH}/config.dist to  ${BACKUP_SCRIPT_PATH}/config before running $0!\n"
+  exit 1
+fi
 
 # import functions
-. /opt/zammad/contrib/backup/functions
+. ${BACKUP_SCRIPT_PATH}/functions
 
 # exec restore
+start_restore_message
+
+get_zammad_dir
+
 restore_warning "${1}"
 
 check_database_config_exists
@@ -28,4 +39,4 @@ restore_zammad
 
 start_zammad
 
-restore_message
+finished_restore_message

contrib/nginx/zammad.conf

@@ -33,6 +33,7 @@ server {
         proxy_set_header Connection "Upgrade";
         proxy_set_header CLIENT_IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
         proxy_read_timeout 86400;
         proxy_pass http://zammad-websocket;
     }
@@ -41,6 +42,7 @@ server {
         proxy_set_header Host $http_host;
         proxy_set_header CLIENT_IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
         proxy_read_timeout 180;
         proxy_pass http://zammad;
 

contrib/nginx/zammad_ssl.conf

@@ -124,6 +124,7 @@ server {
         proxy_set_header Connection "Upgrade";
         proxy_set_header CLIENT_IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
         proxy_read_timeout 86400;
         proxy_pass http://zammad-websocket;
     }
@@ -132,6 +133,7 @@ server {
         proxy_set_header Host $http_host;
         proxy_set_header CLIENT_IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
         proxy_read_timeout 180;
         proxy_pass http://zammad;
 

contrib/travis-ci.org/trigger-docker-compose-build.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+#
+# trigger build of https://github.com/zammad/zammad-docker-compose on https://travis-ci.org/zammad/zammad-docker-compose and upload it to https://hub.docker.com/r/zammad/zammad-docker-compose/
+#
+
+REPO_USER="zammad"
+REPO="zammad-docker-compose"
+BRANCH="master"
+
+if [ ${TRAVIS_BRANCH} == 'stable' ]; then
+  curl -X POST \
+    -H "Content-Type: application/json" \
+    -H "Travis-API-Version: 3" \
+    -H "Accept: application/json" \
+    -H "Authorization: token ${TRAVIS_API_TOKEN}" \
+    -d '{"request": {"branch":"'${BRANCH}'"}}' \
+    "https://api.travis-ci.org/repo/${REPO_USER}%2F${REPO}/requests"
+fi

db/migrate/20120101000010_create_ticket.rb

@@ -62,6 +62,7 @@ class CreateTicket < ActiveRecord::Migration
       t.column :last_contact_at,                  :timestamp, limit: 3,   null: true
       t.column :last_contact_agent_at,            :timestamp, limit: 3,   null: true
       t.column :last_contact_customer_at,         :timestamp, limit: 3,   null: true
+      t.column :last_owner_update_at,             :timestamp, limit: 3,   null: true
       t.column :create_article_type_id,           :integer,               null: true
       t.column :create_article_sender_id,         :integer,               null: true
       t.column :article_count,                    :integer,               null: true
@@ -96,6 +97,7 @@ class CreateTicket < ActiveRecord::Migration
     add_index :tickets, [:last_contact_at]
     add_index :tickets, [:last_contact_agent_at]
     add_index :tickets, [:last_contact_customer_at]
+    add_index :tickets, [:last_owner_update_at]
     add_index :tickets, [:create_article_type_id]
     add_index :tickets, [:create_article_sender_id]
     add_index :tickets, [:created_by_id]

db/migrate/20170403000001_fixed_admin_user_permission_920.rb

@@ -159,7 +159,7 @@ class FixedAdminUserPermission920 < ActiveRecord::Migration
         nulloption: true,
         multiple: false,
         null: false,
-        default: Ticket::State.find_by(name: 'open').id,
+        default: Ticket::State.find_by(default_follow_up: true).id,
         translate: true,
         filter: Ticket::State.by_category(:viewable).pluck(:id),
       },
@@ -249,7 +249,7 @@ class FixedAdminUserPermission920 < ActiveRecord::Migration
         nulloption: false,
         multiple: false,
         null: false,
-        default: Ticket::Priority.find_by(name: '2 normal').id,
+        default: Ticket::Priority.find_by(default_create: true).id,
         translate: true,
       },
       editable: false,

db/migrate/20170529132120_ldap_multi_group_mapping.rb

@@ -0,0 +1,24 @@
+class LdapMultiGroupMapping < ActiveRecord::Migration
+  def up
+
+    # return if it's a new setup
+    return if !Setting.find_by(name: 'system_init_done')
+
+    # load existing LDAP config
+    ldap_config = Setting.get('ldap_config')
+
+    # exit early if no config is present
+    return if ldap_config.blank?
+    return if ldap_config['group_role_map'].blank?
+
+    # loop over group role mapping and check
+    # if we need to migrate to new array structure
+    ldap_config['group_role_map'].each do |source, dest|
+      next if dest.is_a?(Array)
+      ldap_config['group_role_map'][source] = [dest]
+    end
+
+    # store updated
+    Setting.set('ldap_config', ldap_config)
+  end
+end

db/migrate/20170822000001_agend_based_sender_issue_1351.rb

@@ -0,0 +1,14 @@
+class AgendBasedSenderIssue1351 < ActiveRecord::Migration
+  def up
+
+    # return if it's a new setup
+    return if !Setting.find_by(name: 'system_init_done')
+    EmailAddress.all.each { |email_address|
+      begin
+        email_address.save!
+      rescue => e
+        Rails.logger.error "Unable to update EmailAddress.find(#{email_address.id}) '#{email_address.inspect}': #{e.message}"
+      end
+    }
+  end
+end

db/migrate/20170830000001_last_owner_update.rb

@@ -0,0 +1,57 @@
+class LastOwnerUpdate < ActiveRecord::Migration
+  def up
+
+    # return if it's a new setup
+    return if !Setting.find_by(name: 'system_init_done')
+
+    # reset assignment_timeout to prevent unwanted things happen
+    Group.all.each { |group|
+      group.assignment_timeout = nil
+      group.save!
+    }
+
+    add_column :tickets, :last_owner_update_at, :timestamp, limit: 3, null: true
+    add_index :tickets, [:last_owner_update_at]
+    Ticket.reset_column_information
+
+    Scheduler.create_if_not_exists(
+      name: 'Process auto unassign tickets',
+      method: 'Ticket.process_auto_unassign',
+      period: 10.minutes,
+      prio: 1,
+      active: true,
+      updated_by_id: 1,
+      created_by_id: 1,
+    )
+
+    state_ids = Ticket::State.by_category(:work_on).pluck(:id)
+    if state_ids.present?
+      ticket_ids = Ticket.where('tickets.state_id IN (?) AND tickets.owner_id != 1', state_ids).order(created_at: :desc).limit(1000).pluck(:id)
+      ticket_ids.each { |ticket_id|
+        ticket = Ticket.find_by(id: ticket_id)
+        next if !ticket
+        ticket.last_owner_update_at = last_owner_update_at(ticket)
+        ticket.save!
+      }
+    end
+  end
+
+  def last_owner_update_at(ticket)
+    type = History::Type.lookup(name: 'updated')
+    if type
+      object = History::Object.lookup(name: 'Ticket')
+      if object
+        attribute = History::Attribute.lookup(name: 'owner')
+        if attribute
+          history = History.where(o_id: ticket.id, history_type_id: type.id, history_object_id: object.id, history_attribute_id: attribute.id).where.not(id_to: 1).order(created_at: :desc).limit(1)
+          if history.present?
+            return history.first.created_at
+          end
+        end
+      end
+    end
+    return nil if ticket.owner_id == 1
+    ticket.created_at
+  end
+
+end

db/seeds.rb

@@ -3345,7 +3345,7 @@ Macro.create_if_not_exists(
   name: 'Close & Tag as Spam',
   perform: {
     'ticket.state_id' => {
-      value: Ticket::State.find_by(name: 'closed').id,
+      value: Ticket::State.by_category(:closed).first.id,
     },
     'ticket.tags' => {
       operator: 'add',
@@ -3365,8 +3365,6 @@ ticket = Ticket.create(
   group_id: Group.find_by(name: 'Users').id,
   customer_id: User.find_by(login: 'nicole.braun@zammad.org').id,
   title: 'Welcome to Zammad!',
-  state_id: Ticket::State.find_by(name: 'new').id,
-  priority_id: Ticket::Priority.find_by(name: '2 normal').id,
 )
 Ticket::Article.create(
   ticket_id: ticket.id,
@@ -3935,7 +3933,7 @@ ObjectManager::Attribute.add(
     nulloption: true,
     multiple: false,
     null: false,
-    default: Ticket::State.find_by(name: 'open').id,
+    default: Ticket::State.find_by(default_follow_up: true).id,
     translate: true,
     filter: Ticket::State.by_category(:viewable).pluck(:id),
   },
@@ -3953,7 +3951,7 @@ ObjectManager::Attribute.add(
         nulloption: false,
         null: true,
         filter: Ticket::State.by_category(:viewable_customer_new).pluck(:id),
-        default: Ticket::State.find_by(name: 'new').id,
+        default: Ticket::State.find_by(default_create: true).id,
       },
     },
     edit: {
@@ -3966,7 +3964,7 @@ ObjectManager::Attribute.add(
         nulloption: false,
         null: true,
         filter: Ticket::State.by_category(:viewable_customer_edit).pluck(:id),
-        default: Ticket::State.find_by(name: 'open').id,
+        default: Ticket::State.find_by(default_follow_up: true).id,
       },
     },
   },
@@ -4025,7 +4023,7 @@ ObjectManager::Attribute.add(
     nulloption: false,
     multiple: false,
     null: false,
-    default: Ticket::Priority.find_by(name: '2 normal').id,
+    default: Ticket::Priority.find_by(default_create: true).id,
     translate: true,
   },
   editable: false,
@@ -5434,6 +5432,13 @@ Scheduler.create_if_not_exists(
   prio: 1,
   active: true,
 )
+Scheduler.create_if_not_exists(
+  name: 'Process auto unassign tickets',
+  method: 'Ticket.process_auto_unassign',
+  period: 10.minutes,
+  prio: 1,
+  active: true,
+)
 Scheduler.create_if_not_exists(
   name: 'Import OTRS diff load',
   method: 'Import::OTRS.diff_worker',
@@ -5597,7 +5602,7 @@ Trigger.create_or_update(
     },
     'ticket.state_id' => {
       'operator' => 'is not',
-      'value' => Ticket::State.lookup(name: 'closed').id,
+      'value' => Ticket::State.by_category(:closed).first.id,
     },
     'article.type_id' => {
       'operator' => 'is',

lib/core_ext/activesupport/lib/active_support/callbacks.rb

@@ -0,0 +1,33 @@
+module ActiveSupport::Callbacks::ClassMethods
+  # Performs actions on a ActiveSupport model without triggering the given callback.
+  # The parameters are the same as for `skip_callback` and `set_callback`:
+  # http://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html
+  #
+  # Keep in mind that variables defined inside the block are only valid there. If you need
+  # to access one after the block has been processed make sure to initialize the variable
+  # before the block. This is the same behaviour as for all blocks.
+  #
+  # ATTENTION: This is not thread-safe and should not be used in threaded environment.
+  #
+  # @param name [Symbol] The name of the callback like e.g. :save
+  # @param when [Symbol] Indicates the time when the callback should run like e.g. :before
+  # @param method [Symbol] The name of the method that should get disabled e.g. :some_example_method
+  #
+  # @example
+  #  User.without_callback(:create, :after, :avatar_for_email_check) do
+  #    User.create(...)
+  #    'example return value'
+  #  end
+  #  #=> 'example return value'
+  #
+  # @return [optional] Returns the return value of the given block
+  def without_callback(*args)
+    begin
+      skip_callback(*args)
+      result = yield
+    ensure
+      set_callback(*args)
+    end
+    result
+  end
+end

lib/core_ext/activesupport/lib/active_support/logger.rb

@@ -1,29 +0,0 @@
-# This customization provides the possiblity to log exception backtraces via the Rails.logger.
-#
-# @example:
-#  begin
-#    instance = "String :)"
-#    instance.invalid_method
-#  rescue => e
-#    Rails.logger.error e
-#  end
-#  #=> undefined method `invalid_method' for "String :)":String
-#  #   ... backtrace ...
-# https://github.com/rails/rails/blob/308e84e982b940983b4b3d5b41b0b3ac11fbae40/activesupport/lib/active_support/logger.rb#L101
-module ActiveSupport
-  class Logger < ::Logger
-    class SimpleFormatter < ::Logger::Formatter
-      # original behaviour:
-      # rubocop:disable Lint/UnusedMethodArgument, Style/CaseEquality
-      # This method is invoked when a log event occurs
-      def call(severity, timestamp, progname, msg)
-        return "#{String === msg ? msg : msg.inspect}\n" if !msg.is_a?(Exception)
-        # rubocop:enable Lint/UnusedMethodArgument, Style/CaseEquality
-        # custom -> print only the message if no backtrace is present
-        return "#{msg.message}\n" if !msg.backtrace
-        # otherwise combination of message and backtrace
-        "#{msg.message}\n#{msg.backtrace.join("\n")}\n"
-      end
-    end
-  end
-end

lib/core_ext/activesupport/lib/active_support/tagged_logging/formatter.rb

@@ -0,0 +1,30 @@
+# This customization removes the tagged logging functionality in favour of logging exception backtraces via the Rails.logger.
+# Zammad uses Logger::Formatter which partly provides the functionality to log exceptions if given.
+# ActiveSupport::TaggedLogging::Formatter removes this by addind the tags as a string which converts the Exception class to
+# a flat string without the backtrace and other information. It's reduced to only the exception text. This is not wanted
+# in our context.
+# ActiveSupport::TaggedLogging::Formatter addresses:
+# subdomains, request ids, and anything else to aid debugging of multi-user production applications.
+# Which is not needed for us
+#
+# @example:
+#  begin
+#    instance = "String :)"
+#    instance.invalid_method
+#  rescue => e
+#    Rails.logger.error e
+#  end
+#  #=> undefined method `invalid_method' for "String :)":String
+#  #   ... backtrace ...
+# https://github.com/rails/rails/blob/89fab56597c335bb49887563b9a98386b5171574/activesupport/lib/active_support/tagged_logging.rb
+module ActiveSupport
+  module TaggedLogging
+    module Formatter
+      # This method is invoked when a log event occurs.
+      def call(severity, timestamp, progname, msg)
+        # super(severity, timestamp, progname, "#{tags_text}#{msg}")
+        super(severity, timestamp, progname, msg)
+      end
+    end
+  end
+end

lib/html_sanitizer.rb

@@ -19,7 +19,58 @@ satinize html string based on whiltelist
     classes_whitelist = ['js-signatureMarker']
     attributes_2_css = %w(width height)
 
-    scrubber = Loofah::Scrubber.new do |node|
+    scrubber_link = Loofah::Scrubber.new do |node|
+
+      # check if href is different to text
+      if external && node.name == 'a' && !url_same?(node['href'], node.text)
+        if node['href'].blank?
+          node.replace node.children.to_s
+          Loofah::Scrubber::STOP
+        elsif (node.children.empty? || node.children.first.class == Nokogiri::XML::Text) && node.text.present?
+          text = Nokogiri::XML::Text.new("#{node['href']} (", node.document)
+          node.add_previous_sibling(text)
+          node['href'] = cleanup_target(node.text)
+          text = Nokogiri::XML::Text.new(')', node.document)
+          node.add_next_sibling(text)
+        else
+          node.content = cleanup_target(node['href'])
+        end
+      end
+
+      # check if text has urls which need to be clickable
+      if node && node.name != 'a' && node.parent && node.parent.name != 'a' && (!node.parent.parent || node.parent.parent.name != 'a')
+        if node.class == Nokogiri::XML::Text
+          urls = []
+          node.content.scan(%r{((http|https|ftp|tel)://.+?)([[:space:]]|\.[[:space:]]|,[[:space:]]|\.$|,$|\)|\(|$)}mxi).each { |match|
+            if match[0]
+              urls.push match[0].to_s.strip
+            end
+          }
+          node.content.scan(/(^|:|;|\s)(www\..+?)([[:space:]]|\.[[:space:]]|,[[:space:]]|\.$|,$|\)|\(|$)/mxi).each { |match|
+            if match[1]
+              urls.push match[1].to_s.strip
+            end
+          }
+          next if urls.empty?
+          add_link(node.content, urls, node)
+        end
+      end
+
+      # prepare links
+      if node['href']
+        href = cleanup_target(node['href'])
+        if external && href.present? && !href.downcase.start_with?('//') && href.downcase !~ %r{^.{1,6}://.+?}
+          node['href'] = "http://#{node['href']}"
+          href = node['href']
+        end
+        next if !href.downcase.start_with?('http', 'ftp', '//')
+        node.set_attribute('href', href)
+        node.set_attribute('rel', 'nofollow noreferrer noopener')
+        node.set_attribute('target', '_blank')
+      end
+    end
+
+    scrubber_wipe = Loofah::Scrubber.new do |node|
 
       # remove tags with subtree
       if tags_remove_content.include?(node.name)
@@ -38,7 +89,7 @@ satinize html string based on whiltelist
 
       # replace tags, keep subtree
       if !tags_whitelist.include?(node.name)
-        node.replace strict(node.children.to_s)
+        node.replace node.children.to_s
         Loofah::Scrubber::STOP
       end
 
@@ -128,57 +179,19 @@ satinize html string based on whiltelist
           Loofah::Scrubber::STOP
         end
       end
-
-      # prepare links
-      if node['href']
-        href = cleanup_target(node['href'])
-        if external && !href.downcase.start_with?('//') && href.downcase !~ %r{^.{1,6}://.+?}
-          node['href'] = "http://#{node['href']}"
-          href = node['href']
-        end
-        next if !href.downcase.start_with?('http', 'ftp', '//')
-        node.set_attribute('href', href)
-        node.set_attribute('rel', 'nofollow noreferrer noopener')
-        node.set_attribute('target', '_blank')
-      end
-
-      # check if href is different to text
-      if external && node.name == 'a' && !url_same?(node['href'], node.text)
-        if node['href'].blank?
-          node.replace strict(node.children.to_s)
-          Loofah::Scrubber::STOP
-        elsif node.children.empty? || node.children.first.class == Nokogiri::XML::Text
-          text = Nokogiri::XML::Text.new("#{node['href']} (", node.document)
-          node.add_previous_sibling(text)
-          node['href'] = cleanup_target(node.text)
-          text = Nokogiri::XML::Text.new(')', node.document)
-          node.add_next_sibling(text)
-        else
-          node.content = cleanup_target(node['href'])
-        end
-      end
-
-      # check if text has urls which need to be clickable
-      if node && node.name != 'a' && node.parent && node.parent.name != 'a' && (!node.parent.parent || node.parent.parent.name != 'a')
-        if node.class == Nokogiri::XML::Text
-          urls = []
-          node.content.scan(%r{((http|https|ftp|tel)://.+?)([[:space:]]|\.[[:space:]]|,[[:space:]]|\.$|,$|\)|\(|$)}mxi).each { |match|
-            if match[0]
-              urls.push match[0].to_s.strip
-            end
-          }
-          node.content.scan(/(^|:|;|\s)(www\..+?)([[:space:]]|\.[[:space:]]|,[[:space:]]|\.$|,$|\)|\(|$)/mxi).each { |match|
-            if match[1]
-              urls.push match[1].to_s.strip
-            end
-          }
-          next if urls.empty?
-          add_link(node.content, urls, node)
-        end
-      end
-
     end
-    Loofah.fragment(string).scrub!(scrubber).to_s
+
+    new_string = ''
+    done = true
+    while done
+      new_string = Loofah.fragment(string).scrub!(scrubber_wipe).to_s
+      if string == new_string
+        done = false
+      end
+      string = new_string
+    end
+
+    Loofah.fragment(string).scrub!(scrubber_link).to_s
   end
 
 =begin
@@ -253,7 +266,6 @@ cleanup html string:
         Loofah::Scrubber::STOP
       end
     end
-    string = Loofah.fragment(string).scrub!(scrubber_structure).to_s
 
     new_string = ''
     done = true

lib/import/base.rb

@@ -0,0 +1,52 @@
+# Copyright (C) 2012-2016 Zammad Foundation, http://zammad-foundation.org/
+
+module Import
+  class Base
+
+    # Checks if the backend is able to get queued by the Scheduler.
+    #
+    # @example
+    #  Import::ExampleBackend.queueable?
+    #  #=> true
+    #
+    # return [Boolean]
+    def self.queueable?
+      true
+    end
+
+    # Checks if the backend is able to get rescheduled in case the Scheduler
+    # got (re-)started while this ImportJob was running. Defaults to false.
+    #
+    # @example
+    #  instance = Import::LDAP.new(import_job)
+    #  instance.reschedule?(delayed_job)
+    #  #=> false
+    #
+    # return [false]
+    def reschedule?(_delayed_job)
+      false
+    end
+
+    # Initializes a new instance with a stored reference to the ImportJob.
+    #
+    # @example
+    #  instance = Import::ExampleBackend.new(import_job)
+    #
+    # return [Import::ExampleBackend]
+    def initialize(import_job)
+      @import_job = import_job
+    end
+
+    # Starts the life or dry run import of the backend.
+    #
+    # @example
+    #  instance = Import::ExampleBackend.new(import_job)
+    #
+    # @raise [RuntimeError] Raised if the implementation of this mandatory method is missing
+    #
+    # return [nil]
+    def start
+      raise "Missing implementation if the 'start' method."
+    end
+  end
+end

lib/import/base_resource.rb

@@ -5,7 +5,9 @@ module Import
     attr_reader :resource, :remote_id, :errors
 
     def initialize(resource, *args)
+      @action = :unknown
       handle_args(resource, *args)
+      initialize_associations_states
       import(resource, *args)
     end
 
@@ -14,7 +16,7 @@ module Import
     end
 
     def source
-      import_class_namespace
+      self.class.source
     end
 
     def remote_id(resource, *_args)
@@ -23,15 +25,13 @@ module Import
 
     def action
       return :failed if errors.present?
-      return :skipped if !@resource
+      return :skipped if @resource.blank?
       return :unchanged if !attributes_changed?
-      return :created if created?
-      :updated
+      @action
     end
 
     def attributes_changed?
-      return true if changed_attributes.present?
-      @associations_init != associations_state(@resource)
+      changed_attributes.present? || changed_associations.present?
     end
 
     def changed_attributes
@@ -42,16 +42,38 @@ module Import
       @resource.previous_changes
     end
 
-    def created?
-      return false if @resource.blank?
-      # dry run
-      return @resource.created_at.nil? if @resource.changed?
-      # live run
-      @resource.created_at == @resource.updated_at
+    def changed_associations
+      changes = {}
+      tracked_associations.each do |association|
+        # skip if no new value will get assigned (no change is performed)
+        next if !@associations[:after].key?(association)
+        # skip if both values are equal
+        next if @associations[:before][association] == @associations[:after][association]
+        # skip if both values are blank
+        next if @associations[:before][association].blank? && @associations[:after][association].blank?
+        # store changes
+        changes[association] = [@associations[:before][association], @associations[:after][association]]
+      end
+      changes
+    end
+
+    def self.source
+      import_class_namespace
+    end
+
+    def self.import_class_namespace
+      @import_class_namespace ||= name.to_s.sub('Import::', '')
     end
 
     private
 
+    def initialize_associations_states
+      @associations = {}
+      %i(before after).each do |state|
+        @associations[state] ||= {}
+      end
+    end
+
     def import(resource, *args)
       create_or_update(map(resource, *args), *args)
     rescue => e
@@ -69,21 +91,35 @@ module Import
       @resource = lookup_existing(resource, *args)
       return false if !@resource
 
-      # delete since we have an update and
-      # the record is already created
-      resource.delete(:created_by_id)
+      # lock the current resource for write access
+      @resource.with_lock do
 
-      @resource.assign_attributes(resource)
+        # delete since we have an update and
+        # the record is already created
+        resource.delete(:created_by_id)
 
-      # the return value here is kind of misleading
-      # and should not be trusted to indicate if a
-      # resource was actually updated.
-      # Use .action instead
-      return true if !attributes_changed?
+        # store the current state of the associations
+        # from the resource hash because if we assign
+        # them to the instance some (e.g. has_many)
+        # will get stored even in the dry run :/
+        store_associations(:after, resource)
 
-      return true if @dry_run
-      @resource.save
-      true
+        associations = tracked_associations
+        @resource.assign_attributes(resource.except(*associations))
+
+        # the return value here is kind of misleading
+        # and should not be trusted to indicate if a
+        # resource was actually updated.
+        # Use .action instead
+        return true if !attributes_changed?
+
+        @action = :updated
+
+        return true if @dry_run
+        @resource.assign_attributes(resource.slice(*associations))
+        @resource.save!
+        true
+      end
     end
 
     def lookup_existing(resource, *_args)
@@ -96,19 +132,30 @@ module Import
       return if !synced_instance
       instance = import_class.find_by(id: synced_instance.o_id)
 
-      store_associations_state(instance)
+      store_associations(:before, instance)
 
       instance
     end
 
-    def store_associations_state(instance)
-      @associations_init = associations_state(instance)
+    def store_associations(state, source)
+      @associations[state] = associations_state(source)
     end
 
-    def associations_state(instance)
+    def associations_state(source)
       state = {}
       tracked_associations.each do |association|
-        state[association] = instance.send(association)
+        # we have to support instances and (resource) hashes
+        # here since in case of an update we only have the
+        # hash as a source but on create we have an instance
+        if source.is_a?(Hash)
+          # ignore if there is no key for the association
+          # of the Hash (update)
+          # otherwise wrong changes may get detected
+          next if !source.key?(association)
+          state[association] = source[association]
+        else
+          state[association] = source.send(association)
+        end
       end
       state
     end
@@ -132,8 +179,10 @@ module Import
 
     def create(resource, *_args)
       @resource = import_class.new(resource)
+      store_associations(:after, @resource)
+      @action = :created
       return if @dry_run
-      @resource.save
+      @resource.save!
       external_sync_create(
         local:  @resource,
         remote: resource,
@@ -177,11 +226,7 @@ module Import
     end
 
     def mapping_config(*_args)
-      import_class_namespace.gsub('::', '_').underscore + '_mapping'
-    end
-
-    def import_class_namespace
-      self.class.name.to_s.sub('Import::', '')
+      self.class.import_class_namespace.gsub('::', '_').underscore + '_mapping'
     end
 
     def handle_args(_resource, *args)

lib/import/helper.rb

@@ -9,13 +9,16 @@ module Import
       raise 'System is not in import mode!'
     end
 
-    # log
+    def check_system_init_done
+      return true if !Setting.get('system_init_done')
+      raise 'System is already system_init_done!'
+    end
+
     def log(message)
       thread_no = Thread.current[:thread_no] || '-'
       Rails.logger.info "thread##{thread_no}: #{message}"
     end
 
-    # utf8 convert
     def utf8_encode(data)
       data.each { |key, value|
         next if !value

lib/import/ldap.rb

@@ -1,19 +1,55 @@
+# Copyright (C) 2012-2016 Zammad Foundation, http://zammad-foundation.org/
+
 require 'ldap'
 require 'ldap/group'
 
 module Import
-  class Ldap
+  class Ldap < Import::Base
 
-    def initialize(import_job)
-      @import_job = import_job
-
-      if !Setting.get('ldap_integration') && !@import_job.dry_run
-        raise "LDAP integration deactivated, check Setting 'ldap_integration'."
-      end
+    # Checks if the integration is activated and configured.
+    # Otherwise it won't get queued since it will display
+    # an error which is confusing and wrong.
+    #
+    # @example
+    #  Import::LDAP.queueable?
+    #  #=> true
+    #
+    # return [Boolean]
+    def self.queueable?
+      Setting.get('ldap_integration') && Setting.get('ldap_config').present?
+    end
 
+    # Starts a live or dry run LDAP import.
+    #
+    # @example
+    #  instance = Import::LDAP.new(import_job)
+    #
+    # @raise [RuntimeError] Raised if an import should start but the ldap integration is disabled
+    #
+    # return [nil]
+    def start
+      return if !requirements_completed?
       start_import
     end
 
+    # Gets called when the Scheduler gets (re-)started and a LDAP ImportJob was still
+    # in the queue. The job will always get restarted to avoid the gap till the next
+    # run triggered by the Scheduler. The result will get updated to inform the user
+    # in the agent interface result view.
+    #
+    # @example
+    #  instance = Import::LDAP.new(import_job)
+    #  instance.reschedule?(delayed_job)
+    #  #=> true
+    #
+    # return [true]
+    def reschedule?(_delayed_job)
+      @import_job.update_attribute(:result, {
+                                     info: 'Restarting due to scheduler restart.'
+                                   })
+      true
+    end
+
     private
 
     def start_import
@@ -27,5 +63,23 @@ module Import
 
       @import_job.result = Import::Ldap::UserFactory.statistics
     end
+
+    def requirements_completed?
+      return true if @import_job.dry_run
+
+      if !Setting.get('ldap_integration')
+        message = 'Sync cancelled. LDAP integration deactivated. Activate via the switch.'
+      elsif Setting.get('ldap_config').blank? && @import_job.payload.blank?
+        message = 'Sync cancelled. LDAP configration or ImportJob payload missing.'
+      end
+
+      return true if !message
+
+      @import_job.update_attribute(:result, {
+                                     info: message
+                                   })
+
+      false
+    end
   end
 end

lib/import/ldap/user.rb

@@ -6,13 +6,41 @@ module Import
         @remote_id
       end
 
+      def self.lost_map(found_remote_ids)
+        ExternalSync.joins('INNER JOIN users ON (users.id = external_syncs.o_id)')
+                    .where(
+                      source: source,
+                      object: import_class.name,
+                      users:  {
+                        active: true
+                      }
+                    )
+                    .pluck(:source_id, :o_id)
+                    .to_h
+                    .except(*found_remote_ids)
+      end
+
+      def self.deactivate_lost(lost_ids)
+        # we need to update in slices since some DBs
+        # have a limit for IN length
+        lost_ids.each_slice(5000) do |slice|
+
+          # we need to instanciate every entry and set
+          # the active state this way to send notifications
+          # to the client
+          ::User.where(id: slice).each do |user|
+            user.update_attribute(:active, false)
+          end
+        end
+      end
+
       private
 
       def import(resource, *args)
         normalized_entry = normalize_entry(resource)
 
         # extract the uid attribute and store it as
-        # the remote ID so we can access later
+        # the remote ID so we can access it later
         # when working with ExternalSync
         @remote_id = normalized_entry[ @ldap_config[:user_uid].to_sym ]
 
@@ -30,15 +58,26 @@ module Import
       end
 
       def create_or_update(resource, *args)
-        return if skip?(resource)
-        resource[:role_ids] = role_ids(resource)
-        result = super(resource, *args)
+        result = nil
+        if skip?(resource)
+          ldap_log(
+            action:  "skipped -> #{@remote_id}",
+            status:  'success',
+            request: resource,
+          )
+        else
+          catch(:no_roles_assigned) do
+            determine_role_ids(resource)
 
-        ldap_log(
-          action:  "#{action} -> #{@resource.login}",
-          status:  'success',
-          request: resource,
-        )
+            result = super(resource, *args)
+
+            ldap_log(
+              action:  "#{action} -> #{@resource.login}",
+              status:  'success',
+              request: resource,
+            )
+          end
+        end
 
         result
       end
@@ -47,25 +86,49 @@ module Import
         return true if resource[:login].blank?
 
         # skip resource if only ignored attributes are set
-        ignored_attributes = %i(login dn created_by_id updated_by_id)
+        ignored_attributes = %i(login dn created_by_id updated_by_id active)
         !resource.except(*ignored_attributes).values.any?(&:present?)
       end
 
-      def role_ids(resource)
-        # remove remporary added and get value
+      def determine_role_ids(resource)
+        # remove temporary added and get value
         dn = resource.delete(:dn)
-        # use signup roles if no dn is present
-        return @signup_role_ids if !dn
-        # check if roles are mapped for the found dn
-        roles = @dn_roles[ dn.downcase ]
-        # use signup roles if no mapped roles were found
-        return @signup_role_ids if !roles
-        # return found roles
-        roles
+        raise "Missing 'dn' attribute for remote id '#{@remote_id}'" if dn.blank?
+
+        if @dn_roles.present?
+          # check if roles are mapped for the found dn
+          roles = @dn_roles[ dn.downcase ]
+
+          if roles.present?
+            # LDAP is the leading source if
+            # a mapping entry is present
+            @update_role_ids = roles
+            @create_role_ids = roles
+          elsif @ldap_config[:unassigned_users] == 'skip_sync'
+            throw :no_roles_assigned
+          else
+            use_signup_roles
+          end
+        else
+          use_signup_roles
+        end
+      end
+
+      def use_signup_roles
+        @update_role_ids = nil # use existing
+        @create_role_ids = @signup_role_ids
       end
 
       def updated?(resource, *_args)
-        super
+
+        resource[:role_ids] = @update_role_ids if @update_role_ids
+
+        user_found = false
+        import_class.without_callback(:update, :after, :avatar_for_email_check) do
+          user_found = super
+        end
+
+        user_found
       rescue => e
         ldap_log(
           action:   "update -> #{resource[:login]}",
@@ -94,7 +157,7 @@ module Import
             remote: resource,
           )
 
-          store_associations_state(instance)
+          store_associations(:before, instance)
         end
 
         instance
@@ -122,7 +185,10 @@ module Import
       end
 
       def create(resource, *_args)
-        super
+        resource[:role_ids] = @create_role_ids
+        import_class.without_callback(:create, :after, :avatar_for_email_check) do
+          super
+        end
       rescue => e
         ldap_log(
           action:   "create -> #{resource[:login]}",
@@ -143,6 +209,11 @@ module Import
           mapped[attribute] = mapped[attribute].downcase
         end
 
+        # we have to add the active state manually
+        # because otherwise disabled instances won't get
+        # re-activated if they should get synced again
+        mapped[:active] = true
+
         mapped
       end
 
@@ -187,7 +258,6 @@ module Import
           updated_by_id: 1,
         )
       end
-
     end
   end
 end