Deterministic, Zero-Copy Parallel Execution for R

Documentation for package ‘shard’ version 0.1.1

Help Pages

A B C D E F I L M N P Q R S T U V W misc

-- A --

adapter	Adapter Registry for Class-Specific Deep Sharing
affinity	CPU Affinity + mmap Advice (Advanced)
affinity_supported	Check whether CPU affinity is supported
altrep	ALTREP Shared Vectors
arena	Arena Semantic Scope
arena_depth	Get Current Arena Depth
as.array.shard_buffer	Coerce a Shared Memory Buffer to Array
as.double.shard_buffer	Coerce a Shared Memory Buffer to Double
as.integer.shard_buffer	Coerce a Shared Memory Buffer to Integer
as.logical.shard_buffer	Coerce a Shared Memory Buffer to Logical
as.matrix.shard_buffer	Coerce a Shared Memory Buffer to Matrix
as.raw.shard_buffer	Coerce a Shared Memory Buffer to Raw
as.vector.shard_buffer	Coerce a Shared Memory Buffer to a Vector
as_shared	Create a shared vector from an existing R vector
as_tibble	Materialize a shard table handle as a data.frame/tibble
as_tibble.shard_dataset	Materialize a dataset handle into a data.frame/tibble
as_tibble.shard_row_groups	Materialize a row-groups handle into a data.frame/tibble
as_tibble.shard_table_buffer	Materialize a fixed table handle or buffer
as_tibble.shard_table_handle	Materialize a table handle into a data.frame/tibble
attr<-.shard_shared_vector	Set an Attribute on a Shared Vector
attributes<-.shard_shared_vector	Set Attributes on a Shared Vector
available_backings	Get available shared memory backing types

-- B --

bool	Column Types
buffer	Shared Memory Buffers
buffer_advise	Advise access pattern for a buffer
buffer_close	Close a Buffer
buffer_diagnostics	Buffer Diagnostics
buffer_info	Get Buffer Info
buffer_open	Open an Existing Buffer
buffer_path	Get Buffer Path

-- C --

close.shard_deep_shared	Close a Shared Object
close.shard_shared	Close a Shared Object
close.shard_shared_vector	Close a Shared Object
collect	Collect a shard table into memory
collect.shard_dataset	Collect a dataset handle into memory
collect.shard_row_groups	Collect a row-groups handle into memory
collect.shard_table_handle	Collect a table handle into memory
coltypes	Column Types
copy_report	Data Copy Report
cow_report	Copy-on-Write Policy Report

-- D --

diagnostics	Diagnostics API
dim.shard_buffer	Dimensions of a Shared Memory Buffer
dim<-.shard_shared_vector	Set dim on a Shared Vector
dimnames<-.shard_shared_vector	Set dimnames on a Shared Vector
dispatch	Task Dispatch Engine
dispatch_chunks	Dispatch Chunks to Worker Pool

-- E --

ergonomics

Ergonomic Apply/Lapply Wrappers

-- F --

factor_col	Categorical column type
fetch	Fetch Data from a Shared Object
fetch.default	Fetch Data from a Shared Object
fetch.shard_deep_shared	Fetch Data from a Shared Object
fetch.shard_shared	Fetch Data from a Shared Object
float64	Column Types

-- I --

idx_range	Contiguous index range
int32	Column Types
in_arena	Check if Currently Inside an Arena
is_block_view	View Predicates
is_shared	Check if Object is Shared
is_shared_vector	Check if an object is a shared vector
is_view	View Predicates
is_windows	Check if running on Windows
iterate_row_groups	Iterate row groups

-- L --

length.shard_buffer	Length of a Shared Memory Buffer
length.shard_descriptor	Length of a shard_descriptor Object
length.shard_descriptor_lazy	Length of a shard_descriptor_lazy Object
list_kernels	List registered kernels

-- M --

materialize	Materialize Shared Object
materialize.default	Materialize Shared Object
materialize.shard_shared	Materialize Shared Object
materialize.shard_view_block	Materialize a block view into an R matrix
materialize.shard_view_gather	Materialize a gather view into an R matrix
mem_report	Memory Usage Report

-- N --

names<-.shard_shared_vector

Set Names on a Shared Vector

-- P --

pin_workers	Pin shard workers to CPU cores
pool	Worker Pool Management
pool_create	Create a Worker Pool
pool_dispatch	Dispatch Task to Worker
pool_get	Get the Current Worker Pool
pool_health_check	Check Pool Health
pool_lapply	Parallel Dispatch with Async Workers
pool_sapply	Parallel sapply with Supervision
pool_status	Get Pool Status
pool_stop	Stop the Worker Pool
print.arena_result	Print an arena_result object
print.shard_apply_policy	Print a shard_apply_policy Object
print.shard_buffer	Print a Shared Memory Buffer
print.shard_deep_shared	Print a Deep-Shared Object
print.shard_descriptor	Print a shard_descriptor Object
print.shard_descriptor_lazy	Print a shard_descriptor_lazy Object
print.shard_dispatch_result	Print a shard_dispatch_result Object
print.shard_health_report	Print a shard_health_report Object
print.shard_idx_range	Print a shard_idx_range object
print.shard_pool	Print a shard_pool Object
print.shard_reduce_result	Print a shard_reduce_result Object
print.shard_report	Print a shard_report Object
print.shard_result	Print a shard_result Object
print.shard_segment	Print a Shared Memory Segment
print.shard_shared	Print a Shared Object
print.shard_shared_vector	Print a Shared Vector
print.shard_tiles	Print a shard_tiles object
print.shard_view_block	Print a shard_view_block object
print.shard_view_gather	Print a shard_view_gather object
print.shard_worker	Print a shard_worker Object

-- Q --

queue

Chunk Queue Management

-- R --

raw_col	Column Types
recommendations	Performance Recommendations
register_kernel	Register a shard kernel
report	Generate Shard Runtime Report
results	Extract Results from shard_map
row_layout	Row layout for fixed-row table outputs
rss	RSS Monitoring Utilities

-- S --

schema	Define a table schema
scratch_diagnostics	Scratch pool diagnostics
scratch_matrix	Get a scratch matrix
scratch_pool_config	Configure scratch pool limits
segment	Shared Memory Segment
segment_advise	Advise OS about expected access pattern for a segment
segment_close	Close a shared memory segment
segment_create	Create a new shared memory segment
segment_info	Get segment information
segment_open	Open an existing shared memory segment
segment_path	Get the path or name of a segment
segment_protect	Make a segment read-only
segment_read	Read raw data from a segment
segment_report	Shared Memory Segment Report
segment_size	Get the size of a segment
segment_write	Write data to a segment
set_affinity	Set CPU affinity for the current process
shards	Shard Descriptor Creation
shards_list	Create Shards from an Explicit Index List
shard_apply_matrix	Apply a Function Over Matrix Columns with Shared Inputs
shard_apply_policy	Apply Wrapper Policy
shard_crossprod	Parallel crossprod() using shard views + output buffers
shard_get_adapter	Get Adapter for an Object
shard_lapply_shared	Apply a Function Over a List with Optional Auto-Sharing
shard_list_adapters	List Registered Adapters
shard_map	Parallel Execution with shard_map
shard_reduce	Streaming Reductions over Shards
shard_register_adapter	Register an Adapter for Class-Specific Traversal
shard_share_hook	Deep Sharing Hook for Custom Classes
shard_share_hook.default	Deep Sharing Hook for Custom Classes
shard_unregister_adapter	Unregister an Adapter
share	Zero-Copy Shared Objects
shared_advise	Advise access pattern for a shared input vector/matrix
shared_diagnostics	Get diagnostics for a shared vector
shared_info	Get Information About a Shared Object
shared_reset_diagnostics	Reset diagnostic counters for a shared vector
shared_segment	Get the underlying segment from a shared vector
shared_vector	Create a shared vector from a segment
shared_view	Create a view (subset) of a shared vector
share_open	Open an Existing Shared Object by Path
stream_count	Stream row count
stream_filter	Stream-filter a dataset/row-groups into a new partitioned dataset
stream_group_count	Stream group-wise count
stream_group_sum	Stream group-wise sum
stream_map	Stream over row-groups/datasets and map
stream_map.shard_dataset	Stream over row-groups/datasets and map
stream_map.shard_row_groups	Stream over row-groups/datasets and map
stream_reduce	Stream over row-groups/datasets and reduce
stream_reduce.shard_dataset	Stream over row-groups/datasets and reduce
stream_reduce.shard_row_groups	Stream over row-groups/datasets and reduce
stream_sum	Stream sum of a numeric column
stream_top_k	Stream top-k rows by a numeric column
string_col	Column Types
succeeded	Check if shard_map Succeeded

-- T --

table_buffer	Allocate a fixed-row table buffer
table_diagnostics	Table Diagnostics
table_finalize	Finalize a table buffer or sink
table_finalize.shard_table_buffer	Finalize a table buffer
table_finalize.shard_table_sink	Finalize a sink
table_sink	Create a table sink for row-group or partitioned outputs
table_write	Write tabular results into a table buffer or sink
table_write.shard_table_buffer	Write into a table buffer
table_write.shard_table_sink	Write a shard's row-group output
task_report	Task Execution Report

-- U --

utils

Utility Functions

-- V --

view	Create a view over a shared matrix
views	Zero-copy Views
view_block	Create a contiguous block view
view_diagnostics	View diagnostics
view_gather	Create a gather (indexed) view over a shared matrix
view_info	Introspection for a view

-- W --

worker

Individual Worker Control

-- misc --

[.shard_buffer	Extract Buffer Elements
[.shard_descriptor	Subset Shard Descriptor
[.shard_descriptor_lazy	Subset a shard_descriptor_lazy Object
[<-.shard_buffer	Assign to Buffer Elements
[<-.shard_shared_vector	Subset-assign a Shared Vector
[[.shard_descriptor	Get Single Shard
[[.shard_descriptor_lazy	Extract a Single Shard from a shard_descriptor_lazy Object
[[<-.shard_shared_vector	Double-bracket Subset-assign a Shared Vector