ShardSpecs declare and configure shards managed by a Gazette consumer application.
Users generally work with ShardSpecs in YAML form using the
in a analogous way to how Kubernetes resources are managed with
A collection of shards often share configuration. To make them easier
to work with, the
gazctl tool converts to and from a hoisted
YAML map for end-user presentation. Each shard having no specific value for a
configuration property inherits from the
This conversion happens entirely within the tool – ShardSpecs sent to or queried from consumer member processes are full and complete individual specifications.
A common operational pattern is to define ShardSpecs that maintain a one-to-one correspondence with journal partitions of a topic, as is done with the below example. However ShardSpecs are highly configurable and a variety of other patterns are also possible. Labels can be used to attach metadata to ShardSpecs, and applications can interpret labels of the spec to drive customized processing behaviors.
# Create one shard for journals of examples.journalspace.yaml # Compare to ShardSpec for full field definitions. common: # Etcd prefix into which recovery log hints are persisted. hint_prefix: /gazette/hints # Backups of verified recovery log hints, retained for disaster recovery. hint_backups: 2 # Desired number of members which will live-tail the primary's recovery log. hot_standbys: 1 # Upper-bound amount of time during which transactions *may* process messages # before beginning to commit. If input messages stall before reaching this # duration, the transaction may close immediately to minimize latency. max_txn_duration: 1s # Lower-bound amount of time during which a transaction *must* process messages # before beginning to commit. If input messages stall before reaching this # duration, the transaction will block. min_txn_duration: 0s # Prefix of the journal into which each shard's recovery log is recorded. recovery_log_prefix: examples/word-count/recovery-logs # Label common to all shards. labels: - name: common-label value: common-value shards: - id: shard-000 # Label specific to this shard. labels: - name: custom-label sources: - journal: examples/word-count/deltas/part-000 - id: shard-001 sources: - journal: examples/word-count/deltas/part-001 - id: shard-002 sources: - journal: examples/word-count/deltas/part-002 - id: shard-003 sources: - journal: examples/word-count/deltas/part-003
Like JournalSpecs, ShardSpecs retrieved by the
gazctl tool will include their
respective Etcd modification revisions as field
revision within the rendered YAML.
Consult the JournalSpecs documentation for more detail; ShardSpecs work analogously.
Like JournalSpecs, ShardSpecs may be deleted by adding a
delete: true stanza to
the YAML returned by
gazctl and then applying it.
The field may be applied to an individual spec, or to the hoisted
Consult the JournalSpecs documentation for more detail.