2064 lines
64 KiB
Elixir
2064 lines
64 KiB
Elixir
defmodule Oban.Pro.Engines.Smart do
|
||
@moduledoc """
|
||
The `Smart` engine provides advanced concurrency features, enhanced observability, lighter
|
||
weight bulk processing, and provides a foundation for accurate job lifecycle management. As an
|
||
`Oban.Engine`, it is responsible for all non-plugin database interaction, from inserting through
|
||
executing jobs.
|
||
|
||
Major features include:
|
||
|
||
* [Global Concurrency](#module-global-concurrency) — limit the number of concurrent jobs that
|
||
run across _all_ nodes
|
||
* [Rate Limiting](#module-rate-limiting) — limit the number of jobs that execute within a
|
||
window of time
|
||
* [Queue Partitioning](#module-queue-partitioning) — segment a queue so concurrency or rate
|
||
limits apply separately to each partition
|
||
* [Async Tracking](#module-async-tracking) — bundle job acks (completed, cancelled, etc.) to
|
||
minimize transactions and reduce load on the database
|
||
* [Enhanced Unique](#module-enhanced-unique) — enforce job uniqueness with a custom index to
|
||
accelerate inserting unique jobs safely between processes and nodes
|
||
* [Bulk Inserts](#module-bulk-inserts) — automatically batch inserts to prevent hitting
|
||
database limits, reduce data sent over the wire, and respect `unique` options when using
|
||
`Oban.insert_all/2`
|
||
* [Accurate Snooze](#module-accurate-snooze) — differentiate between attempts with errors and
|
||
intentional snoozes
|
||
|
||
## Installation
|
||
|
||
See the [Smart Engine](adoption.md#1-smart-engine) section in the [adoption guide](adoption.md)
|
||
to get started.
|
||
|
||
## Global Concurrency
|
||
|
||
Global concurrency limits the number of concurrent jobs that run across all nodes.
|
||
|
||
Typically the global concurrency limit is `local_limit * num_nodes`. For example, with three
|
||
nodes and a local limit of 10, you'll have a global limit of 30. If a `global_limit` is present,
|
||
and the `local_limit` is omitted, then the `local_limit` falls back to the `global_limit`.
|
||
|
||
The only way to guarantee that all connected nodes will run _exactly one job_ concurrently is to
|
||
set `global_limit: 1`.
|
||
|
||
Here are some examples:
|
||
|
||
```elixir
|
||
# Execute 10 jobs concurrently across all nodes, with up to 10 on a single node
|
||
my_queue: [global_limit: 10]
|
||
|
||
# Execute 10 jobs concurrently, but only 3 jobs on a single node
|
||
my_queue: [local_limit: 3, global_limit: 10]
|
||
|
||
# Execute at most 1 job concurrently
|
||
my_queue: [global_limit: 1]
|
||
```
|
||
|
||
## Rate Limiting
|
||
|
||
Rate limiting controls the number of jobs that execute within a period of time.
|
||
|
||
Rate limiting uses counts for the same queue from all other nodes in the cluster (with or
|
||
without Distributed Erlang). By default, the limiter uses a sliding window over the configured
|
||
period to accurately approximate a limit, though other algorithms are available.
|
||
|
||
Every job execution counts toward the rate limit, regardless of whether the job completes,
|
||
errors, snoozes, etc.
|
||
|
||
Without a modifier, the `rate_limit` period is defined in seconds. However, you can provide a
|
||
`:second`, `:minute`, `:hour` or `:day` modifier to use more intuitive values.
|
||
|
||
* `period: 30` — 30 seconds
|
||
* `period: {1, :minute}` — 60 seconds
|
||
* `period: {2, :minutes}` — 120 seconds
|
||
* `period: {1, :hour}` — 3,600 seconds
|
||
* `period: {1, :day}` —86,400 seconds
|
||
|
||
Here are a few examples:
|
||
|
||
```elixir
|
||
# Execute at most 60 jobs per minute (1 job per second equivalent)
|
||
my_queue: [rate_limit: [allowed: 60, period: {1, :minute}]]
|
||
|
||
# Execute at most 10 jobs per 30 seconds
|
||
my_queue: [rate_limit: [allowed: 10, period: 30]]
|
||
|
||
# Execute at most 10 jobs per minute
|
||
my_queue: [rate_limit: [allowed: 10, period: {1, :minute}]]
|
||
|
||
# Execute at most 1000 jobs per hour
|
||
my_queue: [rate_limit: [allowed: 1000, period: {1, :hour}]]
|
||
```
|
||
|
||
Using larger time periods allows for smoother tracking of rate limits. For example, expressing
|
||
"1 job per second" as "60 jobs per minute" provides the same throughput but reduces the
|
||
granularity of tracking, resulting in more consistent job execution patterns.
|
||
|
||
> #### Understanding Concurrency Limits {: .info}
|
||
>
|
||
> The local, global, or rate limit with the **lowest value** determines how many jobs are executed
|
||
> concurrently. For example, with a `local_limit` of 10 and a `global_limit` of 20, a single node
|
||
> will run 10 jobs concurrently. If that same queue had a `rate_limit` that allowed 5 jobs within
|
||
> a period, then a single node is limited to 5 jobs.
|
||
|
||
### Rate Limiting Algorithms
|
||
|
||
Three rate limiting algorithms are available, each with different trade-offs:
|
||
|
||
* `:sliding_window` (default) — Uses two time buckets with weighted averaging to provide smooth
|
||
rate limiting. Prevents bursting at window boundaries by gradually transitioning between
|
||
periods.
|
||
|
||
* `:fixed_window` — Resets the count when each period expires. Simple and predictable, but can
|
||
allow bursting at window boundaries (e.g., `allowed` jobs at 11:59:59, then `allowed` more at
|
||
12:00:01).
|
||
|
||
* `:token_bucket` — Tokens refill continuously at a rate of `allowed / period` per second.
|
||
Allows controlled bursting up to `allowed` while maintaining the overall rate over time. Ideal
|
||
for APIs that permit short bursts but enforce sustained limits.
|
||
|
||
Specify the algorithm with the `:algorithm` option:
|
||
|
||
```elixir
|
||
# Use fixed windows for simple, predictable resets
|
||
my_queue: [rate_limit: [allowed: 100, period: {1, :minute}, algorithm: :fixed_window]]
|
||
|
||
# Use token bucket for APIs that allow bursting
|
||
my_queue: [rate_limit: [allowed: 60, period: {1, :minute}, algorithm: :token_bucket]]
|
||
```
|
||
|
||
### Weighted Jobs
|
||
|
||
By default, each job consumes one unit of rate limit capacity. For jobs that vary in resource
|
||
usage, you can assign weights so that heavier jobs consume more capacity.
|
||
|
||
There are three ways to assign weights, in order of precedence:
|
||
|
||
1. **Callback** — Define a `weight/1` callback for runtime calculation based on job data
|
||
2. **Job option** — Set the weight when creating a specific job
|
||
3. **Worker option** — Set a default weight for all jobs from a worker
|
||
|
||
For simple rate adjustments, set a default heavier weight for the worker:
|
||
|
||
```elixir
|
||
defmodule MyApp.HeavyWorker do
|
||
use Oban.Pro.Worker, rate: [weight: 10]
|
||
end
|
||
```
|
||
|
||
The default can be overridden by passing an option to the worker's `new/2` function:
|
||
|
||
```elixir
|
||
MyApp.HeavyWorker.new(args, rate: [weight: 5])
|
||
```
|
||
|
||
For variable rate consumption, you can calculate the weight dynamically with the `weight/1`
|
||
callback:
|
||
|
||
```elixir
|
||
defmodule MyApp.BatchWorker do
|
||
use Oban.Pro.Worker
|
||
|
||
@impl Oban.Pro.Worker
|
||
def weight(%{args: args}), do: length(args["records"])
|
||
|
||
@impl Oban.Pro.Worker
|
||
def process(job), do: # ...
|
||
end
|
||
```
|
||
|
||
Jobs without any weight configuration default to a weight of 1. See `Oban.Pro.Worker` for more
|
||
details on the `weight/1` callback.
|
||
|
||
## Queue Partitioning
|
||
|
||
In addition to global and rate limits at the queue level, you can partition a queue so that it's
|
||
treated as multiple queues where concurrency or rate limits apply separately to each partition.
|
||
|
||
Partitions are specified with a list of fields like `:worker`, `:args`, or `:meta`. When
|
||
partitioning by `:args`, choosing specific keys is highly recommended to keep partitioning
|
||
meaningful. Focused partitioning minimizes the amount of data a queue needs to track and
|
||
simplifies job-fetching queries.
|
||
|
||
### Configuring Partitions
|
||
|
||
The partition syntax is identical for global and rate limits (note that you can partition by
|
||
_global or rate_, but not both.)
|
||
|
||
Here are a few examples of viable partitioning schemes:
|
||
|
||
```elixir
|
||
# Partition by worker alone
|
||
partition: :worker
|
||
|
||
# Partition by the `id` and `account_id` from args, ignoring the worker
|
||
partition: [args: [:id, :account_id]]
|
||
|
||
# Partition by worker and the `account_id` key from args
|
||
partition: [:worker, args: :account_id]
|
||
```
|
||
|
||
Remember, take care to minimize partition cardinality by using a few `keys` whenever possible.
|
||
Partitioning based on _every permutation_ of your `args` makes concurrency or rate limits hard to
|
||
reason about and can negatively impact queue performance.
|
||
|
||
### Global Partitioning
|
||
|
||
Global partitioning changes global concurency behavior. Rather than applying a fixed number for
|
||
the queue, it applies to every partition within the queue.
|
||
|
||
Consider the following example:
|
||
|
||
```elixir
|
||
local_limit: 10, global_limit: [allowed: 1, partition: :worker]
|
||
```
|
||
|
||
The queue is configured to run one job per-worker across every node, but only 10 concurrently on a
|
||
single node. That is in contrast to the standard behaviour of `global_limit`, which would override
|
||
the `local_limit` and only allow 1 concurrent job across every node.
|
||
|
||
Alternatively, you could partition by a single key:
|
||
|
||
```elixir
|
||
local_limit: 10, global_limit: [allowed: 1, partition: [args: :tenant_id]]
|
||
```
|
||
|
||
That configures the queue to run one job concurrently across the entire cluster per `tenant_id`.
|
||
|
||
### Partitioning with Burst Mode
|
||
|
||
Global partitioning includes an advanced feature called "burst mode" for global limits
|
||
with partitioning. This feature allows you to maximize throughput by temporarily exceeding
|
||
per-partition global limits when there are available resources.
|
||
|
||
Each global partition is typically restricted to the configured `allowed` value. However, with
|
||
burst mode enabled, the system can intelligently allocate more jobs to each active partition,
|
||
potentially exceeding the per-partition limit while still respecting the overall queue
|
||
concurrency.
|
||
|
||
This is particularly useful when:
|
||
|
||
1. You have many potential partitions but only a few are active at any given time
|
||
2. You want to maximize throughput while maintaining some level of fairness between partitions
|
||
3. You need to ensure your queues aren't sitting idle when capacity is available
|
||
|
||
Here's an example of a queue that will 5 jobs from a single partition concurrently under load,
|
||
but can burst up to 100 for a single partition when there is available capacity:
|
||
|
||
```elixir
|
||
queues: [
|
||
exports: [
|
||
global_limit: [
|
||
allowed: 5,
|
||
burst: true,
|
||
partition: [args: :tenant_id]
|
||
],
|
||
local_limit: 100
|
||
]
|
||
]
|
||
```
|
||
|
||
Bursting _still respects the overall global concurrency limit_. Using the example above, the
|
||
queue can only execute 100 of a given partition's jobs concurrently across all nodes, even if
|
||
there are more available resources.
|
||
|
||
### Rate Limit Partitioning
|
||
|
||
Rate limit partitions operate similarly to global partitions. Rather than limiting all jobs within
|
||
the queue, they limit each partition within the queue.
|
||
|
||
For example, to allow one job per-worker, every ten seconds, across every instance of the `alpha`
|
||
queue in your cluster:
|
||
|
||
```elixir
|
||
local_limit: 10, rate_limit: [allowed: 1, period: 10, partition: :worker]
|
||
```
|
||
|
||
### Partition Keys Cache
|
||
|
||
The Smart engine maintains a cache of available partition keys to optimize performance when
|
||
fetching jobs. The cache has a configurable time-to-live (TTL) which controls how long partition
|
||
keys are remembered before needing to be refreshed from the database.
|
||
|
||
The default TTL is set to `3_000ms`, which is suitable for most applications. You can adjust it
|
||
by adding configuration in your application's config:
|
||
|
||
```elixir
|
||
# In config/config.exs
|
||
config :oban_pro, Oban.Pro.Partition, keys_cache_ttl: 5_000 # 5 seconds
|
||
```
|
||
|
||
A longer TTL reduces database load but might cause the system to be slower to recognize newly
|
||
created partition keys. A shorter TTL ensures more up-to-date partition information at the cost
|
||
of more frequent database queries.
|
||
|
||
The number of partition keys retrieved is based on the queue's `local_limit` multiplied by a
|
||
configurable factor (default `3`). This ensures the query scales appropriately with queue
|
||
capacity while keeping result sets manageable:
|
||
|
||
```elixir
|
||
# In config/config.exs
|
||
config :oban_pro, Oban.Pro.Partition, keys_limit_multiplier: 5
|
||
```
|
||
|
||
Partitions are selected fairly by prioritizing those with the earliest scheduled jobs, ensuring
|
||
work is distributed across active partitions rather than favoring any single partition.
|
||
|
||
## Async Tracking
|
||
|
||
The results of job execution, e.g. `completed`, `cancelled`, etc., are bundled together into a
|
||
single transaction to minimize load on an app's Ecto pool and the database.
|
||
|
||
Bundling updates and reporting them asynchronously dramatically reduces the number of
|
||
transactions per second. However, async bundling introduces a slight lag (up to 5ms) between job
|
||
execution finishing and recording the outcome in the database.
|
||
|
||
Async tracking can be disabled for specific queues with the `ack_async` option:
|
||
|
||
```elixir
|
||
queues: [
|
||
standard: 30,
|
||
critical: [ack_async: false, local_limit: 10]
|
||
]
|
||
```
|
||
|
||
## Enhanced Unique
|
||
|
||
The `Smart` engine uses an alternative mechanism for unique jobs that's designed for speed,
|
||
correctness, scalability, and simplicity. Uniqueness is enforced through a unique index that
|
||
makes insertion entirely safe between processes and nodes, without the use of advisory locks or
|
||
multiple queries.
|
||
|
||
Unlike standard uniqueness, which is only checked as jobs are inserted, the index-backed version
|
||
applies for the job's entire lifetime. That prevents race conditions where a job changes states
|
||
and inadvertently causes a conflict.
|
||
|
||
When conflicts are detected, the conflicted job, i.e. the one already in the database, is
|
||
annotated with `uniq_conflict: true`.
|
||
|
||
> #### Safe Hash for Uniqueness {: .info}
|
||
>
|
||
> To avoid potential hash collisions when using unique jobs with sub-fields in `args`, enable
|
||
> "safe" hashing:
|
||
>
|
||
> ```elixir
|
||
> config :oban_pro, Oban.Pro.Utils, safe_hash: true
|
||
> ```
|
||
>
|
||
> This applies to `uniq_key`, `chain_key`, and `partition_key` values stored in job meta.
|
||
> Note that generated values will not match previous values for configurations using
|
||
> sub-fields in `args`.
|
||
|
||
> #### Period-based Uniqueness Uses Buckets {: .info}
|
||
>
|
||
> To leverage unique indexes, period-based uniqueness snaps timestamps to fixed time buckets
|
||
> rather than using a sliding window. The current time is rounded down to the nearest multiple
|
||
> of the period, and uniqueness is enforced within that bucket.
|
||
>
|
||
> For example, with a period of 15 minutes (`unique: [period: {15, :minutes}]`) and a job
|
||
> inserted at 10:21:00, uniqueness applies from 10:15:00 to 10:30:00. A duplicate job
|
||
> inserted at 10:28:00 would be rejected, but one inserted at 10:31:00 would be allowed
|
||
> because it falls into the next bucket.
|
||
|
||
## Bulk Inserts
|
||
|
||
Where the `Basic` engine requires you to insert unique jobs individually, the `Smart` engine adds
|
||
unique job support to `Oban.insert_all/2`. No additional configuration is necessary—simply use
|
||
`insert_all` instead for unique jobs.
|
||
|
||
```elixir
|
||
Oban.insert_all(lots_of_unique_jobs)
|
||
```
|
||
|
||
Bulk insert also features automatic batching to support inserting an arbitrary number of jobs
|
||
without hitting database limits (PostgreSQL's binary protocol has a limit of 65,535 parameters
|
||
that may be sent in a single call. That presents an upper limit on the number of rows that may be
|
||
inserted at one time.)
|
||
|
||
```elixir
|
||
list_of_args
|
||
|> Enum.map(&MyApp.MyWorker.new/1)
|
||
|> Oban.insert_all()
|
||
```
|
||
|
||
The default batch size for unique jobs is `250`, and `1_000` for non-unique jobs. Regardless, you
|
||
can override with `batch_size`:
|
||
|
||
```elixir
|
||
Oban.insert_all(lots_of_jobs, batch_size: 1500)
|
||
```
|
||
|
||
It's also possible to set a custom timeout for batch inserts:
|
||
|
||
```elixir
|
||
Oban.insert_all(lots_of_jobs, timeout: :infinity)
|
||
```
|
||
|
||
### Skipping Conflicts
|
||
|
||
By default, when a unique conflict is detected during bulk insert, the conflicting job is still
|
||
returned with `conflict?: true` set. This requires updating the existing row to mark it as a
|
||
conflict, which involves row-level locking.
|
||
|
||
For high-throughput scenarios where you don't need information about conflicting jobs, you can
|
||
use `on_conflict: :skip` to bypass locking entirely:
|
||
|
||
```elixir
|
||
Oban.insert_all(lots_of_unique_jobs, on_conflict: :skip)
|
||
```
|
||
|
||
With this option:
|
||
|
||
* Conflicting jobs are silently skipped without any database locks
|
||
* Only newly inserted jobs are returned in the result list
|
||
* Any `replace:` options on individual changesets are ignored
|
||
|
||
This is ideal for "fire and forget" scenarios where you want to insert jobs as fast as possible
|
||
and don't need to track which jobs already existed.
|
||
|
||
### Automatic Spacing
|
||
|
||
When inserting a large batch of jobs that can't all execute immediately, you can automatically
|
||
space them out over time using the `auto_space` option. This schedules each batch at increasing
|
||
intervals, preventing a flood of jobs from overwhelming your queue.
|
||
|
||
```elixir
|
||
Oban.insert_all(lots_of_jobs, batch_size: 1000, auto_space: 60)
|
||
```
|
||
|
||
With `batch_size: 1000` and `auto_space: 60`, jobs are scheduled as follows:
|
||
|
||
* Jobs 1–1000: scheduled immediately
|
||
* Jobs 1001–2000: scheduled 60 seconds from now
|
||
* Jobs 2001–3000: scheduled 120 seconds from now
|
||
* And so on...
|
||
|
||
The `auto_space` value can be an integer (seconds) or a duration tuple:
|
||
|
||
```elixir
|
||
auto_space: 30 # 30 seconds between batches
|
||
auto_space: {1, :minute} # 1 minute between batches
|
||
auto_space: {5, :minutes} # 5 minutes between batches
|
||
```
|
||
|
||
Note that `auto_space` overrides any `scheduled_at` values set on individual changesets.
|
||
|
||
### Per-Batch Transactions
|
||
|
||
By default, all batches are inserted within a single transaction. If any batch fails, all
|
||
previously inserted jobs are rolled back. For scenarios where you'd rather commit successful
|
||
batches even if a later batch fails, use `transaction: :per_batch`:
|
||
|
||
```elixir
|
||
Oban.insert_all(lots_of_jobs, batch_size: 1000, transaction: :per_batch)
|
||
```
|
||
|
||
With this option, each batch is committed independently. If batch 3 fails, jobs from batches 1
|
||
and 2 remain in the database. Note that on failure, the function still raises an exception and
|
||
does not return references to the successfully inserted jobs.
|
||
|
||
## Accurate Snooze
|
||
|
||
Unlike the `Basic` engine which increments `attempts` and `max_attempts`, the Smart engine rolls
|
||
back the `attempt` on snooze. This approach preserves the original `max_attempts` and records a
|
||
`snoozed` count in `meta`. As a result, it's simple to differentiate between "real" attempts and
|
||
snoozes, and backoff calculation remains accurate regardless of snoozing.
|
||
|
||
The following `process/1` function demonstrates checking a job's `meta` for a `snoozed` count:
|
||
|
||
```elixir
|
||
def process(job) do
|
||
case job.meta do
|
||
%{"orig_scheduled_at" => unix_microseconds, "snoozed" => snoozed} ->
|
||
IO.inspect({snoozed, unix_microseconds}, label: "Times snoozed since")
|
||
|
||
_ ->
|
||
# This job has never snoozed before
|
||
end
|
||
end
|
||
```
|
||
"""
|
||
|
||
@behaviour Oban.Engine
|
||
@behaviour Oban.Pro.Handler
|
||
|
||
import Ecto.Query
|
||
import DateTime, only: [utc_now: 0]
|
||
|
||
alias Ecto.{Changeset, Multi}
|
||
alias Oban.{Backoff, Config, Engine, Job, Repo}
|
||
alias Oban.Pro.{Flusher, Handler, Partition, Producer, Unique, Utils}
|
||
alias Oban.Pro.Limiters.{Global, Local, Rate}
|
||
alias Oban.Pro.Stages.Chain
|
||
alias Oban.Pro.Workflow.Schema, as: WorkflowSchema
|
||
|
||
require Logger
|
||
|
||
@type partition ::
|
||
:worker
|
||
| {:args, atom()}
|
||
| {:meta, atom()}
|
||
| [:worker | {:args, atom()} | {:meta, atom()}]
|
||
| [fields: [:worker | :args | :meta], keys: [atom()]]
|
||
|
||
@type period :: pos_integer() | {pos_integer(), unit()}
|
||
|
||
@type global_limit :: pos_integer() | [allowed: pos_integer(), partition: partition()]
|
||
|
||
@type local_limit :: pos_integer()
|
||
|
||
@type rate_limit :: [allowed: pos_integer(), period: period(), partition: partition()]
|
||
|
||
@type unit :: :second | :seconds | :minute | :minutes | :hour | :hours | :day | :days
|
||
|
||
# Module Attributes
|
||
|
||
@ack_tabs Map.new(0..7, &{&1, :"pro_ack_tab_#{&1}"})
|
||
|
||
@drain_uuid "00000000-0000-0000-0000-000000000000"
|
||
|
||
@registry Oban.Registry
|
||
|
||
@lock_codes ~w(deadlock_detected lock_not_available)a
|
||
|
||
@virtual_opts ~w(ack_async queue refresh_interval xact_delay xact_retry xact_timeout updated_at)a
|
||
|
||
@smart_opts Application.compile_env(:oban_pro, __MODULE__, [])
|
||
|
||
@xact_expected_delay Access.get(@smart_opts, :xact_expected_delay, 20)
|
||
@base_batch_size Access.get(@smart_opts, :base_batch_size, 1_000)
|
||
@max_uniq_retries 10
|
||
|
||
# Macros
|
||
|
||
defguardp is_global(producer) when is_map(producer.meta) and is_map(producer.meta.global_limit)
|
||
|
||
defmacrop merge_jsonb(column, map) do
|
||
quote do
|
||
fragment("? || ?", unquote(column), unquote(map))
|
||
end
|
||
end
|
||
|
||
defmacrop merge_meta_array(meta, key) do
|
||
query = """
|
||
jsonb_set(?, '{#{key}}',
|
||
(SELECT jsonb_agg(DISTINCT v)
|
||
FROM jsonb_array_elements_text(
|
||
COALESCE(?->'#{key}', '[]'::jsonb) || COALESCE(EXCLUDED.meta->'#{key}', '[]'::jsonb)
|
||
) AS v))
|
||
"""
|
||
|
||
quote do
|
||
fragment(unquote(query), unquote(meta), unquote(meta))
|
||
end
|
||
end
|
||
|
||
defmacrop try_xact_lock(pref_key, lock_key) do
|
||
quote do
|
||
fragment(
|
||
"pg_try_advisory_xact_lock(?::int, ?::int)",
|
||
unquote(pref_key),
|
||
unquote(lock_key)
|
||
)
|
||
end
|
||
end
|
||
|
||
defmacrop state_in_string(column, string, prefix) do
|
||
quote do
|
||
fragment(
|
||
"? = ANY(regexp_split_to_array(?, ',')::?.oban_job_state[])",
|
||
unquote(column),
|
||
unquote(string),
|
||
unquote(prefix)
|
||
)
|
||
end
|
||
end
|
||
|
||
defmacrop partition_rank(key, priority, scheduled_at, id, off) do
|
||
quote do
|
||
fragment(
|
||
"row_number() OVER (PARTITION BY ? ORDER BY ?, ?, ?) + ?",
|
||
unquote(key),
|
||
unquote(priority),
|
||
unquote(scheduled_at),
|
||
unquote(id),
|
||
unquote(off)
|
||
)
|
||
end
|
||
end
|
||
|
||
defmacrop unnest_partitions(keys, lims, offs) do
|
||
quote do
|
||
fragment(
|
||
"SELECT unnest(?::text[]) AS key, unnest(?::int[]) AS lim, unnest(?::int[]) AS off",
|
||
unquote(keys),
|
||
unquote(lims),
|
||
unquote(offs)
|
||
)
|
||
end
|
||
end
|
||
|
||
# Handler
|
||
|
||
@impl Handler
|
||
def on_start do
|
||
for {_idx, name} <- @ack_tabs do
|
||
:ets.new(name, [:public, :named_table, read_concurrency: true, write_concurrency: true])
|
||
end
|
||
end
|
||
|
||
@impl Handler
|
||
def on_stop, do: :ok
|
||
|
||
# Engine
|
||
|
||
@impl Engine
|
||
def init(%Config{} = conf, [_ | _] = opts) do
|
||
{validate?, opts} = Keyword.pop(opts, :validate, false)
|
||
|
||
{prod_opts, meta_opts} =
|
||
opts
|
||
|> Keyword.put_new(:ack_async, conf.testing == :disabled)
|
||
|> Keyword.split(@virtual_opts)
|
||
|
||
changeset =
|
||
prod_opts
|
||
|> Keyword.put(:name, conf.name)
|
||
|> Keyword.put(:node, conf.node)
|
||
|> Keyword.put(:meta, meta_opts)
|
||
|> Keyword.put_new(:queue, :default)
|
||
|> Keyword.put_new(:started_at, utc_now())
|
||
|> Keyword.put_new(:updated_at, utc_now())
|
||
|> Producer.new()
|
||
|
||
case Changeset.apply_action(changeset, :insert) do
|
||
{:ok, producer} ->
|
||
if validate? do
|
||
{:ok, producer}
|
||
else
|
||
ack_tab =
|
||
producer.queue
|
||
|> :erlang.phash2(map_size(@ack_tabs))
|
||
|> then(&Map.fetch!(@ack_tabs, &1))
|
||
|
||
virt_opts =
|
||
prod_opts
|
||
|> Keyword.take(~w(ack_async refresh_interval xact_delay xact_retry)a)
|
||
|> Map.new()
|
||
|
||
fun =
|
||
fn ->
|
||
conf
|
||
|> Repo.insert!(changeset)
|
||
|> Map.put(:ack_tab, ack_tab)
|
||
|> Map.merge(virt_opts)
|
||
|> put_producer(conf)
|
||
end
|
||
|
||
transaction(conf, fun, virt_opts)
|
||
end
|
||
|
||
{:error, changeset} ->
|
||
{:error, Utils.to_exception(changeset)}
|
||
end
|
||
end
|
||
|
||
@impl Engine
|
||
def refresh(_conf, %Producer{} = producer) do
|
||
# The `Refresher` module handles updating the database rows independently of this callback
|
||
# being triggered. When the queue's producer is stuck in a transaction retry loop it isn't
|
||
# able to handle new messages, including periodic refresh. That would allow the producer
|
||
# record to become outdated, at which point it is subject to erroneous cleanup.
|
||
%{producer | updated_at: utc_now()}
|
||
end
|
||
|
||
@impl Engine
|
||
def shutdown(%Config{} = conf, %Producer{} = producer) do
|
||
monitor_unregister(conf, producer)
|
||
|
||
put_meta(conf, producer, :shutdown, true)
|
||
catch
|
||
_kind, _reason ->
|
||
producer
|
||
|> Producer.update_meta(:paused, true)
|
||
|> Changeset.apply_action!(:update)
|
||
end
|
||
|
||
defp monitor_unregister(conf, producer) do
|
||
parent = self()
|
||
|
||
Task.start(fn ->
|
||
ref = Process.monitor(parent)
|
||
|
||
receive do
|
||
{:DOWN, ^ref, :process, _pid, _reason} ->
|
||
Process.demonitor(ref, [:flush])
|
||
|
||
Registry.delete_meta(@registry, reg_key(conf, producer))
|
||
end
|
||
end)
|
||
end
|
||
|
||
@impl Engine
|
||
def put_meta(%Config{} = conf, %Producer{} = producer, :flush, _any) do
|
||
producer = run_acks(conf, producer)
|
||
|
||
if is_global(producer) do
|
||
Producer
|
||
|> where(uuid: ^producer.uuid)
|
||
|> then(&Repo.update_all(conf, &1, set: [meta: producer.meta, updated_at: utc_now()]))
|
||
end
|
||
|
||
producer
|
||
end
|
||
|
||
def put_meta(%Config{} = conf, %Producer{} = producer, :paused, true) do
|
||
changeset =
|
||
conf
|
||
|> run_acks(producer)
|
||
|> Producer.update_meta(:paused, true)
|
||
|
||
conf
|
||
|> Repo.update!(changeset)
|
||
|> put_producer(conf)
|
||
end
|
||
|
||
def put_meta(%Config{} = conf, %Producer{} = producer, :shutdown, true) do
|
||
changeset =
|
||
conf
|
||
|> run_acks(producer)
|
||
|> Producer.update_meta(%{paused: true, shutdown_started_at: utc_now()})
|
||
|
||
conf
|
||
|> Repo.update!(changeset)
|
||
|> put_producer(conf)
|
||
end
|
||
|
||
def put_meta(%Config{} = conf, %Producer{} = producer, key, value) do
|
||
changeset = Producer.update_meta(producer, key, value)
|
||
|
||
conf
|
||
|> Repo.update!(changeset)
|
||
|> put_producer(conf)
|
||
end
|
||
|
||
@impl Engine
|
||
def check_meta(_conf, %Producer{} = producer, running) do
|
||
jids = for {_, {_, exec}} <- running, do: exec.job.id
|
||
|
||
meta =
|
||
producer.meta
|
||
|> Map.from_struct()
|
||
|> flatten_windows()
|
||
|
||
producer
|
||
|> Map.take(~w(name node queue uuid started_at updated_at)a)
|
||
|> Map.put(:running, jids)
|
||
|> Map.merge(meta)
|
||
end
|
||
|
||
defp flatten_windows(%{rate_limit: %{windows: _}} = meta) do
|
||
put_in(meta.rate_limit.windows, Rate.flatten(meta))
|
||
end
|
||
|
||
defp flatten_windows(meta), do: meta
|
||
|
||
@impl Engine
|
||
def fetch_jobs(_conf, %{meta: %{paused: true}} = prod, _running) do
|
||
{:ok, {prod, []}}
|
||
end
|
||
|
||
def fetch_jobs(_conf, %{meta: %{local_limit: limit}} = prod, running)
|
||
when map_size(running) >= limit do
|
||
{:ok, {prod, []}}
|
||
end
|
||
|
||
def fetch_jobs(%Config{} = conf, %Producer{} = producer, running) do
|
||
acks = get_acks(producer)
|
||
|
||
multi =
|
||
Multi.new()
|
||
|> Multi.put(:acks, acks)
|
||
|> Multi.put(:conf, conf)
|
||
|> Multi.put(:running, running)
|
||
|> Multi.put(:producer, track_acks(acks, producer))
|
||
|> Multi.run(:afh_locks, &take_flush_locks/2)
|
||
|> Multi.run(:all_producers, &all_producers/2)
|
||
|> Multi.run(:ack_ids, &ack_jobs/2)
|
||
|> Multi.run(:afh_ids, &run_flush_handlers/2)
|
||
|> Multi.run(:demand, &check_demand/2)
|
||
|> Multi.run(:jobs, &fetch_jobs/2)
|
||
|> Multi.run(:tracked, &track_jobs(&1, &2, running))
|
||
|
||
case transaction(conf, multi, producer) do
|
||
{:ok, %{ack_ids: ack_ids, jobs: jobs, tracked: producer}} ->
|
||
del_acks(ack_ids, producer)
|
||
|
||
{:ok, {producer, jobs}}
|
||
|
||
{:error, _op, %{postgres: %{code: :unique_violation, detail: detail}}, changes} ->
|
||
clear_unique_violation(changes.conf, detail, ~w(available executing retryable))
|
||
retry_unique_violation(conf, producer, running)
|
||
|
||
{:error, _operation, :xact_lock, _changes} ->
|
||
jittery_sleep()
|
||
|
||
fetch_jobs(conf, producer, running)
|
||
|
||
{:error, _operation, %{postgres: %{code: :deadlock_detected}}, _changes} ->
|
||
jittery_sleep()
|
||
|
||
fetch_jobs(conf, producer, running)
|
||
|
||
{:error, _operation, error, _changes} ->
|
||
raise error
|
||
end
|
||
catch
|
||
:error, %Postgrex.Error{postgres: %{code: :unique_violation, detail: detail}} ->
|
||
clear_unique_violation(conf, detail, Enum.map(Job.states(), &to_string/1))
|
||
retry_unique_violation(conf, producer, running)
|
||
|
||
:error, %Postgrex.Error{postgres: %{code: code}} when code in @lock_codes ->
|
||
jittery_sleep()
|
||
|
||
fetch_jobs(conf, producer, running)
|
||
end
|
||
|
||
defp retry_unique_violation(conf, producer, running) do
|
||
attempt = Process.get(:fetch_jobs, 0)
|
||
|
||
if attempt >= @max_uniq_retries do
|
||
Process.delete(:fetch_jobs)
|
||
|
||
Logger.error(fn ->
|
||
"[Oban.Pro.Engines.Smart] Unique violation retry limit (#{@max_uniq_retries}) exceeded " <>
|
||
"while fetching jobs. This may indicate a persistent unique constraint conflict that " <>
|
||
"can't be automatically resolved."
|
||
end)
|
||
|
||
{:ok, {producer, []}}
|
||
else
|
||
Process.put(:fetch_jobs, attempt + 1)
|
||
|
||
fetch_jobs(conf, producer, running)
|
||
end
|
||
end
|
||
|
||
@doc false
|
||
def fetch_for_drain(conf, %{queue: queue, with_limit: limit} = opts) do
|
||
subset_query =
|
||
if queue == :__all__ do
|
||
where(Job, state: "available")
|
||
else
|
||
where(Job, state: "available", queue: ^to_string(queue))
|
||
end
|
||
|
||
subset_query =
|
||
case opts do
|
||
%{batch_ids: ids} -> where(subset_query, [j], j.meta["batch_id"] in ^ids)
|
||
%{workflow_ids: ids} -> where(subset_query, [j], j.meta["workflow_id"] in ^ids)
|
||
_ -> subset_query
|
||
end
|
||
|
||
subset_query =
|
||
subset_query
|
||
|> order_by(asc: :priority, asc: :scheduled_at, asc: :id)
|
||
|> lock("FOR UPDATE SKIP LOCKED")
|
||
|> limit(^limit)
|
||
|
||
query =
|
||
Job
|
||
|> with_cte("subset", as: ^subset_query)
|
||
|> join(:inner, [j], x in fragment(~s("subset")), on: true)
|
||
|> where([j, x], j.id == x.id)
|
||
|> select([j, _], j)
|
||
|
||
updates = [
|
||
set: [state: "executing", attempted_at: utc_now(), attempted_by: [conf.node, @drain_uuid]],
|
||
inc: [attempt: 1]
|
||
]
|
||
|
||
{_count, jobs} = Repo.update_all(conf, query, updates, [])
|
||
|
||
jobs
|
||
end
|
||
|
||
@impl Engine
|
||
def stage_jobs(%Config{} = conf, queryable, opts) do
|
||
limit = Keyword.fetch!(opts, :limit)
|
||
|
||
subquery =
|
||
queryable
|
||
|> select([:id, :state])
|
||
|> where([j], j.state in ~w(scheduled retryable))
|
||
|> where([j], not is_nil(j.queue))
|
||
|> where([j], j.scheduled_at <= ^DateTime.utc_now())
|
||
|> limit(^limit)
|
||
|
||
query =
|
||
Job
|
||
|> join(:inner, [j], x in subquery(subquery), on: j.id == x.id)
|
||
|> select([j, x], %{id: j.id, queue: j.queue, state: x.state})
|
||
|
||
{_count, staged} = Repo.update_all(conf, query, set: [state: "available"])
|
||
|
||
{:ok, staged}
|
||
catch
|
||
:error, %Postgrex.Error{postgres: %{code: :unique_violation, detail: detail}} ->
|
||
# Staging runs inside the Stager's transaction, and this violation has already aborted
|
||
# it, so the conflict can't be repaired on this connection. Repair out-of-band and skip
|
||
# this round; the next staging pass promotes the freed jobs.
|
||
Utils.async_maybe_await(fn ->
|
||
clear_unique_violation(conf, detail, ["scheduled", "retryable"])
|
||
end)
|
||
|
||
{:ok, []}
|
||
|
||
:error, %Postgrex.Error{postgres: %{code: :deadlock_detected}} ->
|
||
jittery_sleep()
|
||
|
||
stage_jobs(conf, queryable, opts)
|
||
end
|
||
|
||
@impl Engine
|
||
def check_available(%Config{} = conf) do
|
||
exists =
|
||
Job
|
||
|> where([j], j.state == "available" and parent_as(:producers).queue == j.queue)
|
||
|> select(1)
|
||
|
||
query =
|
||
from(p in Producer, as: :producers)
|
||
|> where([p], exists(subquery(exists)))
|
||
|> select([p], p.queue)
|
||
|
||
{:ok, Repo.all(conf, query)}
|
||
end
|
||
|
||
@impl Engine
|
||
def complete_job(%Config{} = conf, %Job{} = job) do
|
||
set = [state: "completed", completed_at: utc_now()]
|
||
|
||
case Process.get(:oban_meta_update) do
|
||
nil -> put_ack(conf, %{job | state: "completed"}, set)
|
||
map -> put_ack(conf, %{job | state: "completed"}, set ++ [meta: map])
|
||
end
|
||
|
||
:ok
|
||
end
|
||
|
||
@impl Engine
|
||
def discard_job(%Config{} = conf, %Job{} = job) do
|
||
put_ack(conf, %{job | state: "discarded"},
|
||
state: "discarded",
|
||
discarded_at: utc_now(),
|
||
error: Job.format_attempt(job)
|
||
)
|
||
|
||
:ok
|
||
end
|
||
|
||
@impl Engine
|
||
def error_job(%Config{} = conf, %Job{} = job, seconds) do
|
||
orig_at = Map.get(job.meta, "orig_scheduled_at", to_unix(job.scheduled_at))
|
||
|
||
set =
|
||
if job.attempt >= job.max_attempts do
|
||
[state: "discarded", discarded_at: utc_now()]
|
||
else
|
||
[state: "retryable", scheduled_at: seconds_from_now(seconds)]
|
||
end
|
||
|
||
put_ack(
|
||
conf,
|
||
%{job | state: set[:state]},
|
||
set ++ [error: Job.format_attempt(job), meta: %{orig_scheduled_at: orig_at}]
|
||
)
|
||
|
||
:ok
|
||
end
|
||
|
||
@impl Engine
|
||
def snooze_job(%Config{} = conf, %Job{} = job, seconds) do
|
||
snoozed = Map.get(job.meta, "snoozed", 0)
|
||
orig_at = Map.get(job.meta, "orig_scheduled_at", to_unix(job.scheduled_at))
|
||
|
||
meta = %{orig_scheduled_at: orig_at, snoozed: snoozed + 1}
|
||
|
||
meta =
|
||
case Process.get(:oban_meta_update) do
|
||
nil -> meta
|
||
map -> Map.merge(meta, map)
|
||
end
|
||
|
||
put_ack(conf, %{job | state: "scheduled"},
|
||
attempt_change: -1,
|
||
state: "scheduled",
|
||
scheduled_at: seconds_from_now(seconds),
|
||
meta: meta
|
||
)
|
||
|
||
:ok
|
||
end
|
||
|
||
@impl Engine
|
||
def cancel_job(%Config{} = conf, %Job{unsaved_error: %{}} = job) do
|
||
put_ack(conf, %{job | state: "cancelled"},
|
||
state: "cancelled",
|
||
cancelled_at: utc_now(),
|
||
error: Job.format_attempt(job)
|
||
)
|
||
end
|
||
|
||
def cancel_job(%Config{} = conf, %Job{} = job) do
|
||
cancel_all_jobs(conf, where(Job, id: ^job.id))
|
||
|
||
:ok
|
||
end
|
||
|
||
@impl Engine
|
||
def cancel_all_jobs(%Config{} = conf, queryable) do
|
||
subquery = where(queryable, [j], j.state not in ["cancelled", "completed", "discarded"])
|
||
|
||
query =
|
||
Job
|
||
|> join(:inner, [j], x in subquery(subquery), on: j.id == x.id)
|
||
|> update(set: [state: "cancelled", cancelled_at: ^utc_now()])
|
||
|> select([_, x], map(x, [:id, :attempted_by, :meta, :queue, :state, :worker]))
|
||
|
||
{:ok, %{jobs: {_count, jobs}}} =
|
||
Multi.new()
|
||
|> Multi.put(:conf, conf)
|
||
|> Multi.update_all(:jobs, query, [], Repo.default_options(conf))
|
||
|> Multi.run(:ack, &track_cancelled_jobs/2)
|
||
|> then(&Repo.transaction(conf, &1))
|
||
|
||
{:ok, jobs}
|
||
end
|
||
|
||
@doc false
|
||
# This is used to unblock a uniqueness violation by clearing out the uniq_bmp field. The
|
||
# The unique constraint (either column-based or expression-based) uses `uniq_key` from meta,
|
||
# but is only enforced when the job's state maps to a value in `uniq_bmp`.
|
||
#
|
||
# We find conflicting jobs by matching the unique key in meta and filtering by the expected
|
||
# states. Clearing their `uniq_bmp` disables the unique constraint, allowing the transition.
|
||
def clear_unique_violation(%Config{} = conf, detail, states) do
|
||
# Match both column-based `(uniq_key)=(value)` and expression-based `((meta->>'uniq_key'))=(value)`
|
||
# formats. The exact spacing is critical.
|
||
pattern = ~r/\((?:uniq_key|\(meta ->> 'uniq_key'::text\))\)=\((?<key>.+)\)/
|
||
|
||
with %{"key" => key} <- Regex.named_captures(pattern, detail) do
|
||
query =
|
||
Job
|
||
|> where([j], j.state in ^states)
|
||
|> where([j], fragment("? @> ?", j.meta, ^%{uniq_key: key}))
|
||
|> update([j], set: [meta: merge_jsonb(j.meta, ^%{uniq_bmp: []})])
|
||
|
||
Repo.update_all(conf, query, [])
|
||
|
||
:telemetry.execute(
|
||
[:oban, :engine, :uniq_violation_repaired],
|
||
%{count: 1},
|
||
%{conf: conf, uniq_key: key, states: states}
|
||
)
|
||
|
||
log_key = {__MODULE__, :uniq_violation_logged, key}
|
||
|
||
if !:persistent_term.get(log_key, false) do
|
||
:persistent_term.put(log_key, true)
|
||
|
||
Logger.warning(fn ->
|
||
"""
|
||
[Oban.Pro.Engines.Smart] Unique constraint violation repaired.
|
||
|
||
This may indicate a unique job misconfiguration where jobs are being inserted
|
||
with conflicting unique keys across different states. The engine repaired the
|
||
violation for uniq_key: #{key}
|
||
|
||
Check your unique job configuration to ensure the :states option matches the
|
||
expected job lifecycle for this worker.
|
||
"""
|
||
end)
|
||
end
|
||
end
|
||
end
|
||
|
||
@impl Engine
|
||
def insert_job(conf, changeset, opts) do
|
||
if changeset.valid? do
|
||
case insert_all_jobs(conf, [changeset], opts) do
|
||
[job] ->
|
||
{:ok, job}
|
||
|
||
_ ->
|
||
{:error, changeset}
|
||
end
|
||
else
|
||
{:error, changeset}
|
||
end
|
||
end
|
||
|
||
@impl Engine
|
||
def insert_all_jobs(conf, changesets, opts) do
|
||
if opts[:transaction] == :per_batch do
|
||
insert_all_batches(conf, changesets, opts)
|
||
else
|
||
fun = fn -> insert_all_batches(conf, changesets, opts) end
|
||
|
||
{:ok, jobs} = Repo.transaction(conf, fun, opts)
|
||
|
||
jobs
|
||
end
|
||
end
|
||
|
||
defp insert_all_batches(conf, changesets, opts) do
|
||
batch_size = Keyword.get(opts, :batch_size, @base_batch_size)
|
||
auto_space = Keyword.get(opts, :auto_space)
|
||
|
||
changesets
|
||
|> Stream.map(&Unique.with_uniq_meta/1)
|
||
|> Stream.map(&Partition.with_partition_meta(&1, conf))
|
||
|> Stream.uniq_by(&Unique.get_key(&1, System.unique_integer()))
|
||
|> Stream.chunk_every(batch_size)
|
||
|> Stream.with_index()
|
||
|> Stream.map(&apply_auto_space(&1, auto_space))
|
||
|> Enum.flat_map(fn changesets ->
|
||
{uniq_map, link_map, replace?} =
|
||
Enum.reduce(changesets, {%{}, %{}, false}, fn changeset, {uniq_map, link_map, replace?} ->
|
||
uniq_map =
|
||
case Unique.get_key(changeset) do
|
||
uniq_key when is_binary(uniq_key) ->
|
||
Map.put(uniq_map, uniq_key, changeset)
|
||
|
||
_ ->
|
||
uniq_map
|
||
end
|
||
|
||
link_map =
|
||
case Chain.get_key(changeset) do
|
||
chain_id when is_binary(chain_id) ->
|
||
Map.update(link_map, chain_id, [changeset], &[changeset | &1])
|
||
|
||
_ ->
|
||
link_map
|
||
end
|
||
|
||
{uniq_map, link_map, replace? or Unique.replace?(changeset)}
|
||
end)
|
||
|
||
{:ok, %{all_jobs: jobs}} =
|
||
Multi.new()
|
||
|> Multi.put(:conf, conf)
|
||
|> Multi.put(:opts, opts)
|
||
|> Multi.put(:changesets, changesets)
|
||
|> Multi.put(:uniq_map, uniq_map)
|
||
|> Multi.put(:link_map, link_map)
|
||
|> Multi.put(:replacements?, replace?)
|
||
|> Multi.run(:uniq_mode, &uniq_mode/2)
|
||
|> Multi.run(:workflow_conflicts, &insert_workflows/2)
|
||
|> Multi.run(:chain_changesets, &prepare_chains/2)
|
||
|> Multi.run(:dupe_map, &find_dupes/2)
|
||
|> Multi.run(:new_jobs, &insert_entries/2)
|
||
|> Multi.run(:rep_jobs, &apply_replacements/2)
|
||
|> Multi.run(:all_jobs, &apply_conflicts/2)
|
||
|> then(&Repo.transaction(conf, &1, opts))
|
||
|
||
jobs
|
||
end)
|
||
end
|
||
|
||
defp apply_auto_space({changesets, _index}, nil), do: changesets
|
||
|
||
defp apply_auto_space({changesets, batch_index}, auto_space) do
|
||
space_seconds = Utils.cast_period(auto_space)
|
||
scheduled_at = DateTime.add(utc_now(), batch_index * space_seconds, :second)
|
||
|
||
Enum.map(changesets, &Changeset.force_change(&1, :scheduled_at, scheduled_at))
|
||
end
|
||
|
||
defp uniq_mode(_repo, %{conf: conf}) do
|
||
%{name: name, prefix: prefix} = conf
|
||
|
||
Utils.persistent_cache({__MODULE__, :uniq_mode, name}, fn ->
|
||
query =
|
||
from("pg_indexes")
|
||
|> put_query_prefix("pg_catalog")
|
||
|> where(schemaname: ^prefix, tablename: "oban_jobs", indexname: "oban_jobs_unique_index")
|
||
|> where([i], fragment("? LIKE 'CREATE UNIQUE%'", i.indexdef))
|
||
|> select([i], i.indexdef)
|
||
|
||
mode =
|
||
case Repo.one(conf, query) do
|
||
nil -> :none
|
||
indexdef -> if indexdef =~ ~r/\(uniq_key(?:\s+ASC)?\)/, do: :column, else: :expression
|
||
end
|
||
|
||
{:ok, mode}
|
||
end)
|
||
end
|
||
|
||
defp insert_workflows(_repo, %{changesets: changesets, conf: conf}) do
|
||
{unique_workflows, regular_workflows} =
|
||
changesets
|
||
|> Enum.filter(&workflow_changeset?/1)
|
||
|> Enum.group_by(&get_workflow_id/1)
|
||
|> Enum.map(&WorkflowSchema.from_changesets/1)
|
||
|> Enum.split_with(&Changeset.get_field(&1, :meta)[:unique])
|
||
|
||
{existing_ids, taken_names} = lock_workflows(unique_workflows, conf)
|
||
|
||
{old_unique, new_unique} =
|
||
Enum.split_with(
|
||
unique_workflows,
|
||
&MapSet.member?(existing_ids, Changeset.get_field(&1, :id))
|
||
)
|
||
|
||
{conflicting, insertable} =
|
||
Enum.split_with(new_unique, fn changeset ->
|
||
name = Changeset.get_field(changeset, :name)
|
||
name != nil and MapSet.member?(taken_names, name)
|
||
end)
|
||
|
||
Enum.each(regular_workflows ++ old_unique ++ insertable, &upsert_workflow(conf, &1))
|
||
|
||
conflicting_ids =
|
||
conflicting
|
||
|> Enum.map(&Changeset.get_field(&1, :id))
|
||
|> MapSet.new()
|
||
|
||
{:ok, conflicting_ids}
|
||
end
|
||
|
||
defp lock_workflows([], _conf), do: {MapSet.new(), MapSet.new()}
|
||
|
||
defp lock_workflows(workflows, conf) do
|
||
ids = Enum.map(workflows, &Changeset.get_field(&1, :id))
|
||
|
||
names =
|
||
workflows
|
||
|> Enum.map(&Changeset.get_field(&1, :name))
|
||
|> Enum.reject(&is_nil/1)
|
||
|
||
query =
|
||
WorkflowSchema
|
||
|> where([w], w.id in ^ids)
|
||
|> or_where(
|
||
[w],
|
||
w.name in ^names and w.state != "completed" and fragment("meta \\? 'unique'")
|
||
)
|
||
|> select([w], {w.id, w.name})
|
||
|> lock("FOR UPDATE")
|
||
|
||
existing = Repo.all(conf, query)
|
||
|
||
existing_ids = existing |> Enum.map(&elem(&1, 0)) |> MapSet.new()
|
||
|
||
taken_names =
|
||
existing
|
||
|> Enum.map(&elem(&1, 1))
|
||
|> Enum.reject(&is_nil/1)
|
||
|> MapSet.new()
|
||
|
||
{existing_ids, taken_names}
|
||
end
|
||
|
||
defp workflow_changeset?(changeset) do
|
||
changeset
|
||
|> Changeset.get_field(:meta, %{})
|
||
|> is_map_key(:workflow_id)
|
||
end
|
||
|
||
defp get_workflow_id(changeset) do
|
||
changeset
|
||
|> Changeset.get_field(:meta, %{})
|
||
|> Map.get(:workflow_id)
|
||
end
|
||
|
||
defp upsert_workflow(conf, changeset) do
|
||
on_conflict =
|
||
from(wf in WorkflowSchema,
|
||
update: [
|
||
set: [
|
||
suspended: fragment("? + EXCLUDED.suspended", wf.suspended),
|
||
available: fragment("? + EXCLUDED.available", wf.available),
|
||
scheduled: fragment("? + EXCLUDED.scheduled", wf.scheduled),
|
||
executing: fragment("? + EXCLUDED.executing", wf.executing),
|
||
retryable: fragment("? + EXCLUDED.retryable", wf.retryable),
|
||
completed: fragment("? + EXCLUDED.completed", wf.completed),
|
||
cancelled: fragment("? + EXCLUDED.cancelled", wf.cancelled),
|
||
discarded: fragment("? + EXCLUDED.discarded", wf.discarded),
|
||
meta: merge_meta_array(merge_meta_array(wf.meta, "queues"), "workers")
|
||
]
|
||
]
|
||
)
|
||
|
||
Repo.insert(conf, changeset, conflict_target: :id, on_conflict: on_conflict)
|
||
end
|
||
|
||
defp prepare_chains(_repo, %{link_map: link_map, conf: conf}) when map_size(link_map) > 0 do
|
||
chain_ids = Map.keys(link_map)
|
||
|
||
chain_ids
|
||
|> Enum.map(&:erlang.phash2({conf.prefix, :chain, &1}))
|
||
|> take_advisory_locks(conf)
|
||
|
||
query =
|
||
from(
|
||
f in fragment("json_array_elements_text(?)", ^chain_ids),
|
||
as: :list,
|
||
inner_lateral_join:
|
||
j in subquery(
|
||
Job
|
||
|> select([:state])
|
||
|> where([j], j.state != "completed")
|
||
|> where([j], fragment("? \\? 'chain_id'", j.meta))
|
||
|> where([j], fragment("?->>'chain_id'", j.meta) == parent_as(:list).value)
|
||
|> order_by(desc: :id)
|
||
|> lock("FOR UPDATE")
|
||
|> limit(1)
|
||
),
|
||
on: true,
|
||
select: {f.value, j.state}
|
||
)
|
||
|
||
found_map =
|
||
conf
|
||
|> Repo.all(query)
|
||
|> Map.new()
|
||
|
||
linked =
|
||
Enum.flat_map(link_map, fn {chain_id, changesets} ->
|
||
changesets
|
||
|> Enum.reverse()
|
||
|> Enum.with_index()
|
||
|> Enum.map(fn {changeset, index} ->
|
||
state = Map.get(found_map, chain_id)
|
||
meta = Changeset.get_field(changeset, :meta)
|
||
|
||
cond do
|
||
index == 0 and is_nil(state) -> changeset
|
||
index == 0 and Chain.continuable?(state, meta) -> changeset
|
||
true -> Chain.to_suspended(changeset)
|
||
end
|
||
end)
|
||
end)
|
||
|
||
{:ok, linked}
|
||
end
|
||
|
||
defp prepare_chains(_repo, _changes), do: {:ok, []}
|
||
|
||
@impl Engine
|
||
def retry_job(%Config{} = conf, %Job{id: id}) do
|
||
retry_all_jobs(conf, where(Job, [j], j.id == ^id))
|
||
|
||
:ok
|
||
end
|
||
|
||
@impl Engine
|
||
def retry_all_jobs(%Config{} = conf, queryable) do
|
||
subquery = where(queryable, [j], j.state not in ~w(available executing suspended))
|
||
|
||
query =
|
||
Job
|
||
|> join(:inner, [j], x in subquery(subquery), on: j.id == x.id)
|
||
|> select([_, x], map(x, [:id, :queue, :state]))
|
||
|> update([j],
|
||
set: [
|
||
state: "available",
|
||
max_attempts: fragment("GREATEST(?, ? + 1)", j.max_attempts, j.attempt),
|
||
scheduled_at: ^utc_now(),
|
||
completed_at: nil,
|
||
cancelled_at: nil,
|
||
discarded_at: nil
|
||
]
|
||
)
|
||
|
||
{_, jobs} = Repo.update_all(conf, query, [])
|
||
|
||
{:ok, jobs}
|
||
end
|
||
|
||
@impl Engine
|
||
def update_job(%Config{} = conf, %Job{id: id}, changes) when is_map(changes) do
|
||
updater = fn ->
|
||
query =
|
||
Job
|
||
|> where([j], j.id == ^id)
|
||
|> lock("FOR UPDATE SKIP LOCKED")
|
||
|
||
case Repo.one(conf, query) do
|
||
nil ->
|
||
{:error, :locked_or_not_found}
|
||
|
||
job when is_map_key(job.meta, "structured") and is_map_key(changes, :args) ->
|
||
{:ok, worker} = Oban.Worker.from_string(job.worker)
|
||
|
||
changeset = worker.new(changes.args)
|
||
|
||
if changeset.valid? do
|
||
job
|
||
|> Job.update(changes)
|
||
|> then(&Repo.update(conf, &1))
|
||
else
|
||
{:error, changeset}
|
||
end
|
||
|
||
job ->
|
||
job
|
||
|> Job.update(changes)
|
||
|> then(&Repo.update(conf, &1))
|
||
end
|
||
end
|
||
|
||
case Repo.transaction(conf, updater) do
|
||
{:ok, result} -> result
|
||
{:error, reason} -> {:error, reason}
|
||
end
|
||
end
|
||
|
||
# Producer Fetching Helpers
|
||
|
||
defp get_producer(conf, job_or_producer) do
|
||
case Registry.meta(@registry, reg_key(conf, job_or_producer)) do
|
||
{:ok, producer} -> producer
|
||
:error -> %Producer{ack_async: false}
|
||
end
|
||
end
|
||
|
||
defp put_producer(producer, conf) do
|
||
Registry.put_meta(@registry, reg_key(conf, producer), producer)
|
||
|
||
producer
|
||
end
|
||
|
||
defp reg_key(%{name: name}, %{queue: queue}), do: {name, {:producer, queue}}
|
||
|
||
# Acking Helpers
|
||
|
||
defp put_ack(conf, job, updates) do
|
||
producer = get_producer(conf, job)
|
||
global_key = Partition.get_key(job, "*")
|
||
flush_handlers = Flusher.get_flush_handlers(job, conf)
|
||
|
||
ack_entry = {{:ack, producer.name, job.queue, job.id}, global_key, flush_handlers, updates}
|
||
|
||
if producer.ack_tab, do: :ets.insert(producer.ack_tab, ack_entry)
|
||
|
||
cond do
|
||
Process.get(:oban_draining) ->
|
||
Process.delete(:oban_meta_update)
|
||
|
||
{:ok, jids} = ack_jobs([ack_entry], conf)
|
||
|
||
run_flush_handlers([ack_entry], conf)
|
||
|
||
if producer.ack_tab, do: del_acks(jids, producer)
|
||
|
||
not producer.ack_async or producer.meta.paused ->
|
||
pid = Oban.Registry.whereis(conf.name, {:producer, producer.queue})
|
||
|
||
if is_pid(pid) and Process.alive?(pid) do
|
||
GenServer.call(pid, {:put_meta, :flush, true})
|
||
end
|
||
|
||
true ->
|
||
:ok
|
||
end
|
||
end
|
||
|
||
defp get_acks(%{ack_tab: tab, name: name, queue: queue}) do
|
||
:ets.select(tab, [{{{:ack, name, queue, :_}, :_, :_, :_}, [], [:"$_"]}])
|
||
end
|
||
|
||
defp del_acks(ids, %{ack_tab: tab, name: name, queue: queue}) do
|
||
Enum.each(ids, &:ets.delete(tab, {:ack, name, queue, &1}))
|
||
end
|
||
|
||
defp run_acks(conf, producer) do
|
||
acks = get_acks(producer)
|
||
|
||
{:ok, jids} = ack_jobs(acks, conf)
|
||
|
||
run_flush_handlers(acks, conf)
|
||
del_acks(jids, producer)
|
||
track_acks(acks, producer)
|
||
end
|
||
|
||
defp take_flush_locks(_repo, %{acks: acks, conf: conf}) do
|
||
acks
|
||
|> Enum.flat_map(&elem(&1, 2))
|
||
|> Enum.flat_map(fn
|
||
{Oban.Pro.Workflow, _, [wids, _con]} when is_list(wids) -> wids
|
||
_ -> []
|
||
end)
|
||
|> Enum.map(&:erlang.phash2({conf.prefix, :flush, &1}))
|
||
|> take_advisory_locks(conf)
|
||
end
|
||
|
||
defp run_flush_handlers(_repo, %{acks: acks, conf: conf}) do
|
||
{:ok, run_flush_handlers(acks, conf)}
|
||
end
|
||
|
||
defp run_flush_handlers(acks, conf) do
|
||
mfas = for {_, _, all, _} <- acks, mfa <- all, is_tuple(mfa), uniq: true, do: mfa
|
||
|
||
span(:flush, conf, %{count: length(mfas)}, fn ->
|
||
Enum.each(mfas, fn {mod, fun, arg} -> apply(mod, fun, arg) end)
|
||
|
||
{:ok, length(mfas)}
|
||
end)
|
||
end
|
||
|
||
# Fetch Helpers
|
||
|
||
defp all_producers(_repo, %{conf: conf, producer: producer}) do
|
||
%{ack_tab: tab, meta: meta, queue: queue} = producer
|
||
|
||
needs_lock? = is_map(meta.global_limit) or is_map(meta.rate_limit) or any_flush_handlers?(tab)
|
||
query = where(Producer, queue: ^queue)
|
||
|
||
cond do
|
||
not needs_lock? ->
|
||
{:ok, []}
|
||
|
||
not has_advisory_locks?(conf) ->
|
||
{:ok, Repo.all(conf, lock(query, "FOR UPDATE NOWAIT"))}
|
||
|
||
try_advisory_lock?(queue, conf) ->
|
||
{:ok, Repo.all(conf, query)}
|
||
|
||
true ->
|
||
{:error, :xact_lock}
|
||
end
|
||
end
|
||
|
||
defp check_demand(_repo, %{conf: conf, producer: producer} = changes) do
|
||
span(:demand, conf, %{queue: producer.queue}, fn ->
|
||
{:ok, local} = Local.check(changes)
|
||
{:ok, global} = Global.check(changes)
|
||
{:ok, rate} = Rate.check(changes)
|
||
|
||
{:ok, %{local: local, global: global, rate: rate}}
|
||
end)
|
||
end
|
||
|
||
defp any_flush_handlers?(tab) do
|
||
match = {:_, :_, :"$1", :_}
|
||
guard = [{:"/=", :"$1", []}]
|
||
|
||
:ets.select_count(tab, [{match, guard, [true]}]) > 0
|
||
end
|
||
|
||
defp has_advisory_locks?(conf) do
|
||
Utils.persistent_cache({__MODULE__, :has_advisory_locks?}, fn ->
|
||
query =
|
||
from("pg_proc")
|
||
|> put_query_prefix("pg_catalog")
|
||
|> where(proname: "pg_try_advisory_xact_lock", pronargs: 2)
|
||
|> select(true)
|
||
|
||
Repo.one(conf, query) == true
|
||
end)
|
||
end
|
||
|
||
defp try_advisory_lock?(queue, conf) do
|
||
pre = :erlang.phash2(conf.prefix)
|
||
key = :erlang.phash2(queue)
|
||
|
||
query = from(f in try_xact_lock(^pre, ^key), select: f.f0)
|
||
|
||
Repo.one(conf, query)
|
||
end
|
||
|
||
defp take_advisory_locks(keys, conf) do
|
||
if Enum.any?(keys) and has_advisory_locks?(conf) do
|
||
keys = :lists.usort(keys)
|
||
|
||
Repo.query!(conf, "SELECT pg_advisory_xact_lock(unnest($1::bigint[]))", [keys])
|
||
|
||
{:ok, length(keys)}
|
||
else
|
||
{:ok, 0}
|
||
end
|
||
end
|
||
|
||
defp fetch_jobs(_repo, %{conf: conf, producer: producer} = changes) do
|
||
span(:fetch, conf, %{queue: producer.queue}, fn ->
|
||
subset_query = fetch_subquery(changes)
|
||
|
||
query =
|
||
Job
|
||
|> with_cte("subset", as: ^subset_query)
|
||
|> join(:inner, [j], x in fragment(~s("subset")), on: true)
|
||
|> where([j, x], j.id == x.id and j.state == "available" and j.attempt < j.max_attempts)
|
||
|> select([j, _], j)
|
||
|
||
updates = [
|
||
set: [
|
||
state: "executing",
|
||
attempted_at: utc_now(),
|
||
attempted_by: [conf.node, producer.uuid]
|
||
],
|
||
inc: [attempt: 1]
|
||
]
|
||
|
||
{_count, jobs} = Repo.update_all(conf, query, updates)
|
||
|
||
{:ok, jobs}
|
||
end)
|
||
end
|
||
|
||
defp fetch_subquery(%{demand: %{local: local, global: global, rate: rate}, producer: producer}) do
|
||
case {global, rate} do
|
||
{nil, nil} ->
|
||
fetch_subquery(producer, local)
|
||
|
||
{global, nil} when is_integer(global) ->
|
||
fetch_subquery(producer, min(local, global))
|
||
|
||
{nil, rate} when is_integer(rate) ->
|
||
fetch_subquery(producer, min(local, rate))
|
||
|
||
{global, %{} = rate_demands} ->
|
||
fetch_subquery(producer, {rate_demands, %{}}, min(local, global || local))
|
||
|
||
{{_lims, _offs} = demands, rate} ->
|
||
fetch_subquery(producer, demands, min(local, rate || local))
|
||
|
||
{global, rate} ->
|
||
limit = rate |> min(local) |> min(global)
|
||
|
||
fetch_subquery(producer, limit)
|
||
end
|
||
end
|
||
|
||
defp fetch_subquery(producer, limit) do
|
||
Job
|
||
|> select([:id])
|
||
|> where(state: "available", queue: ^producer.queue)
|
||
|> order_by([:priority, :scheduled_at, :id])
|
||
|> limit(^max(limit, 0))
|
||
|> lock("FOR UPDATE SKIP LOCKED")
|
||
end
|
||
|
||
defp fetch_subquery(producer, {lims_map, offs_map}, limit) do
|
||
{keys, lims, offs} =
|
||
Enum.reduce(lims_map, {[], [], []}, fn {key, lim}, {keys, lims, offs} ->
|
||
{[key | keys], [lim | lims], [Map.get(offs_map, key, 0) | offs]}
|
||
end)
|
||
|
||
part_query =
|
||
from(
|
||
p in unnest_partitions(^keys, ^lims, ^offs),
|
||
as: :part,
|
||
inner_lateral_join:
|
||
j in subquery(
|
||
Job
|
||
|> where([j], j.state == "available" and j.queue == ^producer.queue)
|
||
|> where([j], fragment("meta \\? 'partition_key'"))
|
||
|> where([j], fragment("meta->>'partition_key'") == parent_as(:part).key)
|
||
|> order_by([:priority, :scheduled_at, :id])
|
||
|> limit(parent_as(:part).lim)
|
||
|> select([j], %{
|
||
id: j.id,
|
||
priority: j.priority,
|
||
scheduled_at: j.scheduled_at,
|
||
key: parent_as(:part).key,
|
||
off: parent_as(:part).off
|
||
})
|
||
),
|
||
on: true,
|
||
select: %{
|
||
id: j.id,
|
||
priority: j.priority,
|
||
scheduled_at: j.scheduled_at,
|
||
rank: partition_rank(j.key, j.priority, j.scheduled_at, j.id, j.off)
|
||
}
|
||
)
|
||
|
||
from(j in subquery(part_query),
|
||
select: j.id,
|
||
limit: ^limit,
|
||
order_by: [asc: j.rank, asc: j.priority, asc: j.scheduled_at, asc: j.id]
|
||
)
|
||
end
|
||
|
||
# Tracking Helpers
|
||
|
||
@ack_fields ~w(state cancelled_at completed_at discarded_at scheduled_at attempt_change error meta)a
|
||
|
||
@doc false
|
||
# This is public for sharing with chunk.ex, which needs to ack chunk jobs _without_ tracking
|
||
# them as normally executed jobs. The initial caluse will emit `fetch_jobs` telemetry, and is
|
||
# purposefully limited to calls in the fetch multi.
|
||
def ack_jobs(_repo, %{acks: acks, conf: conf}) do
|
||
span(:ack, conf, %{count: length(acks)}, fn -> ack_jobs(acks, conf) end)
|
||
end
|
||
|
||
def ack_jobs([], _conf), do: {:ok, []}
|
||
|
||
def ack_jobs(acks, conf) do
|
||
[ids | params] =
|
||
acks
|
||
|> Enum.map(fn {{:ack, _, _, id}, _, _, set} -> [id | Enum.map(@ack_fields, &set[&1])] end)
|
||
|> Enum.zip_with(&Function.identity/1)
|
||
|
||
case Repo.query(conf, ack_query(conf.prefix), [ids | params]) do
|
||
{:ok, %{rows: rows}} -> {:ok, List.flatten(rows)}
|
||
error -> error
|
||
end
|
||
end
|
||
|
||
defp ack_query(prefix) do
|
||
Utils.persistent_cache({__MODULE__, :ack_query, prefix}, fn ->
|
||
"""
|
||
WITH params AS (
|
||
SELECT unnest($1::bigint[]) AS id,
|
||
unnest($2::"#{prefix}".oban_job_state[]) AS state,
|
||
unnest($3::timestamp[]) AS cancelled_at,
|
||
unnest($4::timestamp[]) AS completed_at,
|
||
unnest($5::timestamp[]) AS discarded_at,
|
||
unnest($6::timestamp[]) AS scheduled_at,
|
||
unnest($7::integer[]) AS attempt_change,
|
||
unnest($8::jsonb[]) AS error,
|
||
unnest($9::jsonb[]) AS meta
|
||
),
|
||
locked AS (
|
||
SELECT oj.id
|
||
FROM "#{prefix}"."oban_jobs" oj
|
||
INNER JOIN params tmp ON oj.id = tmp.id
|
||
FOR UPDATE OF oj
|
||
)
|
||
UPDATE "#{prefix}"."oban_jobs" oj
|
||
SET state = tmp.state,
|
||
cancelled_at = COALESCE(tmp.cancelled_at, oj.cancelled_at),
|
||
completed_at = COALESCE(tmp.completed_at, oj.completed_at),
|
||
discarded_at = COALESCE(tmp.discarded_at, oj.discarded_at),
|
||
scheduled_at = CASE
|
||
WHEN (tmp.meta ? 'wait_until') AND (oj.meta ? 'signal')
|
||
THEN oj.scheduled_at
|
||
ELSE COALESCE(tmp.scheduled_at, oj.scheduled_at)
|
||
END,
|
||
attempt = CASE
|
||
WHEN tmp.attempt_change IS NULL THEN oj.attempt
|
||
WHEN oj.state = 'executing' THEN GREATEST(tmp.attempt_change + oj.attempt, 0)
|
||
ELSE oj.attempt
|
||
END,
|
||
errors = CASE WHEN tmp.error IS NULL THEN oj.errors ELSE oj.errors || tmp.error END,
|
||
meta = CASE WHEN tmp.meta IS NULL THEN oj.meta ELSE oj.meta || tmp.meta END
|
||
FROM params tmp
|
||
INNER JOIN locked l ON tmp.id = l.id
|
||
WHERE oj.id = tmp.id
|
||
RETURNING oj.id
|
||
"""
|
||
end)
|
||
end
|
||
|
||
defp track_jobs(_repo, %{conf: conf, jobs: jobs, producer: producer}, running) do
|
||
old_jobs = for {_ref, {_pid, %{job: job}}} <- running, do: job
|
||
all_jobs = jobs ++ old_jobs
|
||
|
||
meta =
|
||
producer.meta
|
||
|> Global.track(all_jobs)
|
||
|> Local.track(jobs)
|
||
|> Rate.track(jobs)
|
||
|
||
if meta == producer.meta do
|
||
{:ok, producer}
|
||
else
|
||
now = utc_now()
|
||
query = where(Producer, uuid: ^producer.uuid)
|
||
|
||
case Repo.update_all(conf, query, set: [meta: meta, updated_at: now]) do
|
||
{1, _} ->
|
||
{:ok, %{producer | meta: meta, updated_at: now}}
|
||
|
||
# In this case the producer was erroneously deleted, possibly due to a connection error,
|
||
# downtime, or in development after waking from sleep.
|
||
{0, _} ->
|
||
%{producer | meta: meta, updated_at: now}
|
||
|> Changeset.change()
|
||
|> then(&Repo.insert(conf, &1))
|
||
end
|
||
end
|
||
end
|
||
|
||
defp track_acks(acks, producer) when is_global(producer) do
|
||
keys = Enum.map(acks, &elem(&1, 1))
|
||
meta = Global.prepare_tracked(producer.meta, keys)
|
||
|
||
%{producer | meta: meta}
|
||
end
|
||
|
||
defp track_acks(_acks, producer), do: producer
|
||
|
||
defp track_cancelled_jobs(_repo, %{conf: conf, jobs: {_count, jobs}}) do
|
||
jobs
|
||
|> Enum.filter(&(&1.state == "executing"))
|
||
|> Enum.group_by(& &1.attempted_by)
|
||
|> Enum.each(fn {[_node, uuid | _], jobs} ->
|
||
query =
|
||
Producer
|
||
|> where([p], p.uuid == ^uuid)
|
||
|> where([p], fragment("?->'global_limit' \\? 'tracked'", p.meta))
|
||
|
||
with %Producer{meta: meta} = producer <- Repo.one(conf, query) do
|
||
keys = Enum.map(jobs, &Partition.get_key(&1, "*"))
|
||
meta = Global.prepare_tracked(meta, keys)
|
||
|
||
%{producer | meta: meta, updated_at: utc_now()}
|
||
|> Changeset.change()
|
||
|> then(&Repo.update(conf, &1))
|
||
end
|
||
end)
|
||
|
||
{:ok, nil}
|
||
end
|
||
|
||
# Insert Helpers
|
||
|
||
defp find_dupes(_repo, %{conf: conf, uniq_mode: :none, uniq_map: uniq_map})
|
||
when map_size(uniq_map) > 0 do
|
||
{uniq_keys, uniq_states} =
|
||
Enum.reduce(uniq_map, {[], []}, fn {key, changeset}, {key_acc, sta_acc} ->
|
||
states = changeset |> Unique.get_states() |> Enum.join(",")
|
||
|
||
{[key | key_acc], [states | sta_acc]}
|
||
end)
|
||
|
||
query =
|
||
from(
|
||
t in fragment(
|
||
"SELECT unnest(?::text[]) AS uk, unnest(?::text[]) AS us",
|
||
^uniq_keys,
|
||
^uniq_states
|
||
),
|
||
join: j in Job,
|
||
on:
|
||
fragment("? \\? 'uniq_key'", j.meta) and
|
||
fragment("?->>'uniq_key'", j.meta) == t.uk and
|
||
state_in_string(j.state, t.us, literal(^conf.prefix)),
|
||
select: {t.uk, j.id, j.state}
|
||
)
|
||
|
||
dupes =
|
||
conf
|
||
|> Repo.all(query)
|
||
|> Map.new(fn {key, id, state} -> {key, {id, state, Map.get(uniq_map, key)}} end)
|
||
|
||
{:ok, dupes}
|
||
end
|
||
|
||
defp find_dupes(_repo, _changes), do: {:ok, %{}}
|
||
|
||
defp insert_entries(_repo, %{changesets: []}), do: {:ok, []}
|
||
|
||
defp insert_entries(_repo, changes) do
|
||
%{conf: conf, changesets: changesets, chain_changesets: chains} = changes
|
||
%{dupe_map: dupe_map, opts: opts, workflow_conflicts: workflow_conflicts} = changes
|
||
|
||
changesets =
|
||
if chains == [] do
|
||
changesets
|
||
else
|
||
Enum.reject(changesets, &Chain.chain?/1) ++ chains
|
||
end
|
||
|
||
{entries, placeholders} =
|
||
for changeset <- changesets,
|
||
not is_map_key(dupe_map, Unique.get_key(changeset)),
|
||
not workflow_conflicted?(changeset, workflow_conflicts),
|
||
reduce: {[], nil} do
|
||
{entries, acc} ->
|
||
map = Job.to_map(changeset)
|
||
|
||
if is_nil(acc) do
|
||
{[map | entries], map}
|
||
else
|
||
{[map | entries], Map.filter(acc, fn {key, val} -> map[key] == val end)}
|
||
end
|
||
end
|
||
|
||
# Ensure placeholders aren't nil before use in is_map_key.
|
||
placeholders = placeholders || %{}
|
||
|
||
entries =
|
||
for entry <- Enum.reverse(entries) do
|
||
Map.new(entry, fn {key, val} ->
|
||
if is_map_key(placeholders, key) do
|
||
{key, {:placeholder, key}}
|
||
else
|
||
{key, val}
|
||
end
|
||
end)
|
||
end
|
||
|
||
opts =
|
||
opts
|
||
|> Keyword.merge(placeholders: placeholders, returning: true)
|
||
|> Keyword.merge(conflict_opts(changes))
|
||
|
||
{_count, jobs} = Repo.insert_all(conf, Job, entries, opts)
|
||
|
||
{:ok, jobs}
|
||
end
|
||
|
||
defp conflict_opts(%{uniq_map: uniq_map, uniq_mode: mode, opts: opts, conf: conf}) do
|
||
cond do
|
||
uniq_map == %{} or mode == :none ->
|
||
[on_conflict: :nothing]
|
||
|
||
opts[:on_conflict] == :skip ->
|
||
[
|
||
conflict_target: {:unsafe_fragment, conflict_target(mode, conf)},
|
||
on_conflict: :nothing
|
||
]
|
||
|
||
true ->
|
||
[
|
||
conflict_target: {:unsafe_fragment, conflict_target(mode, conf)},
|
||
on_conflict: update(Job, [j], set: [meta: merge_jsonb(j.meta, ^%{uniq_conflict: true})])
|
||
]
|
||
end
|
||
end
|
||
|
||
defp conflict_target(:column, _conf), do: "(uniq_key) WHERE uniq_key IS NOT NULL"
|
||
|
||
defp conflict_target(:expression, conf) do
|
||
"""
|
||
((meta->>'uniq_key')) WHERE meta ? 'uniq_key' AND jsonb_contains(meta->'uniq_bmp', #{conf.prefix}.oban_state_to_bit(state))
|
||
"""
|
||
end
|
||
|
||
defp apply_replacements(_repo, %{conf: conf, opts: opts} = changes) do
|
||
cond do
|
||
not changes.replacements? ->
|
||
{:ok, []}
|
||
|
||
changes.opts[:on_conflict] == :skip ->
|
||
{:ok, []}
|
||
|
||
true ->
|
||
updates =
|
||
for {_key, {id, state, %{changes: %{replace: replace} = changes}}} <- dupe_map(changes),
|
||
reduce: %{} do
|
||
acc ->
|
||
state = String.to_existing_atom(state)
|
||
rep_keys = Keyword.get(replace, state, [])
|
||
|
||
changes
|
||
|> Map.take(rep_keys)
|
||
|> Enum.reduce(acc, fn {key, val}, sub_acc ->
|
||
Map.update(sub_acc, {key, val}, [id], &[id | &1])
|
||
end)
|
||
end
|
||
|
||
Enum.each(updates, fn {val, ids} ->
|
||
Repo.update_all(conf, where(Job, [j], j.id in ^ids), [set: [val]], opts)
|
||
end)
|
||
|
||
{:ok, []}
|
||
end
|
||
end
|
||
|
||
defp dupe_map(%{dupe_map: dupe_map, uniq_mode: :none}), do: dupe_map
|
||
|
||
defp dupe_map(%{new_jobs: new_jobs, uniq_map: uniq_map}) do
|
||
Enum.reduce(new_jobs, %{}, fn job, acc ->
|
||
case job do
|
||
%{meta: %{"uniq_conflict" => true, "uniq_key" => uniq_key}} ->
|
||
Map.put(acc, uniq_key, {job.id, job.state, Map.get(uniq_map, uniq_key)})
|
||
|
||
_ ->
|
||
acc
|
||
end
|
||
end)
|
||
end
|
||
|
||
defp apply_conflicts(_repo, %{uniq_mode: :none} = changes) do
|
||
%{dupe_map: dupe_map, new_jobs: new_jobs} = changes
|
||
|
||
old_jobs =
|
||
dupe_map
|
||
|> Map.values()
|
||
|> Enum.map(fn {_id, _state, changeset} ->
|
||
changeset
|
||
|> Changeset.apply_action!(:insert)
|
||
|> Map.replace!(:conflict?, true)
|
||
end)
|
||
|
||
workflow_conflict_jobs = apply_workflow_conflicts(changes)
|
||
|
||
{:ok, old_jobs ++ workflow_conflict_jobs ++ new_jobs}
|
||
end
|
||
|
||
defp apply_conflicts(_repo, changes) do
|
||
%{new_jobs: new_jobs} = changes
|
||
|
||
new_jobs =
|
||
Enum.map(new_jobs, fn
|
||
%{meta: %{"uniq_conflict" => true}} = job -> %{job | conflict?: true}
|
||
job -> job
|
||
end)
|
||
|
||
workflow_conflict_jobs = apply_workflow_conflicts(changes)
|
||
|
||
{:ok, workflow_conflict_jobs ++ new_jobs}
|
||
end
|
||
|
||
defp apply_workflow_conflicts(%{changesets: changesets, workflow_conflicts: conflicts}) do
|
||
changesets
|
||
|> Enum.filter(&workflow_conflicted?(&1, conflicts))
|
||
|> Enum.map(fn changeset ->
|
||
changeset
|
||
|> Changeset.apply_action!(:insert)
|
||
|> Map.replace!(:conflict?, true)
|
||
end)
|
||
end
|
||
|
||
defp workflow_conflicted?(changeset, workflow_conflicts) do
|
||
workflow_id = get_workflow_id(changeset)
|
||
|
||
workflow_id != nil and MapSet.member?(workflow_conflicts, workflow_id)
|
||
end
|
||
|
||
# Telemetry Helpers
|
||
|
||
defp span(event, conf, meta, fun) do
|
||
meta = Map.put(meta, :conf, conf)
|
||
|
||
:telemetry.span([:oban, :engine, :fetch_jobs, event], meta, fn ->
|
||
case fun.() do
|
||
{:ok, result} -> {{:ok, result}, Map.put(meta, :result, result)}
|
||
{:error, _} = error -> {error, meta}
|
||
end
|
||
end)
|
||
end
|
||
|
||
# Time Helpers
|
||
|
||
defp seconds_from_now(seconds), do: DateTime.add(utc_now(), seconds, :second)
|
||
|
||
defp to_unix(datetime), do: DateTime.to_unix(datetime, :microsecond)
|
||
|
||
# Xact Helpers
|
||
|
||
defp jittery_sleep do
|
||
@xact_expected_delay
|
||
|> Backoff.jitter()
|
||
|> Process.sleep()
|
||
end
|
||
|
||
defp transaction(conf, fun_or_multi, prod_opts) do
|
||
opts = [
|
||
delay: Map.get(prod_opts, :xact_delay, 1000),
|
||
retry: Map.get(prod_opts, :xact_retry, 5),
|
||
timeout: Map.get(prod_opts, :xact_timeout, :timer.seconds(30)),
|
||
expected_delay: @xact_expected_delay,
|
||
expected_retry: 500
|
||
]
|
||
|
||
Repo.transaction(conf, fun_or_multi, opts)
|
||
end
|
||
end
|