No articles match
dataprep: a leakage-free preprocessing workflow14 days ago
The data-leakage problem | A minimal example | What is stored in the plan | A note on degenerate columns | Adding or removing steps | Column-order independence | Missing-column detection | Workflow with dataprep() | Reporting | Full applied workflow | When NOT to preprocess | Test environment | Where to go next | Session info
dataprep: cleaning pipeline14 days ago
Quick start | Overview | Why obsedele changed in 0.1.8 | Boundary behaviour | condextr vs percoutl | Why obsedele is re-applied after condextr | Step-by-step walkthrough | Step 1 — Variable deletion | Step 2 — Observation deletion | Before cleaning | Step 3 — Conditional extremum outlier removal | After cleaning | Step 4 — Short-period interpolation | One-call pipeline: dataprep() | Inspecting the plan without running it | Missing-value diagnostics | Additional cleaning helpers | When NOT to preprocess | Where to go next | Session info
dataprep: descriptive statistics and diagnostic plots14 days ago
Overview | Descriptive statistics | Line plot, numeric variable names | Selected statistics | Bar chart, character variable names | Control facet layout | Full-size data | The underlying table | Percentile plots | Full percentile range | Top percentiles only | Bottom percentiles only | Percentile curves of the raw data | Numeric axis control | Combining with ggplot2 | A diagnostic workflow | Where to go next | Session info
dataprep: design philosophy and preprocessing methodology14 days ago
The problem this package solves | The four-step pipeline | Variable deletion: three principles | 1. Signal-to-noise ratio as the primary threshold | 2. Physical relevance | 3. Missing-run length, not just missing fraction | Observation deletion: two principles | 1. Detect the pattern, don't filter the points | 2. The half-hour window is the physical mixing time | How it is implemented: anchor-based scan | Implementation evolution | Why "both sides" and not "one side"? | Subsets: group and period | Re-applied after outlier deletion | Outlier removal: two principles | 1. Per-point decisions, not one-shot thresholds | 2. Outlier deletion inherits the gap-length constraint | Missing-value interpolation: two principles | 1. Vertical, not horizontal | 2. Segment the series before interpolating | Why this order, and not any other | Evaluation metrics | Number of values deleted | Sample retention | Residual outliers | Newly created outliers | Uncertainty analysis | The package beyond the four steps | Related documentation | Funding | Session info
dataprep: fast reshaping with melt() and dcast()14 days ago
Why another reshape implementation | Test environment | Wide to long with melt() | Basic usage | Measure-side specification | Automatic ID inference | Custom column names and na.rm | Memory layout: major = "row" vs major = "col" | Thread control | Why melt() is fast | 1. Two layout paths, selected by major | 2. SIMD streaming stores | 3. Hugepage hint for large outputs | 4. Fast paths for small inputs | 5. Cache-aware block sizing | 6. Per-call caches for SEXP and factor levels | Long to wide with dcast() | Formula interface | Fill missing cells | Aggregating duplicate pairs | na.rm | Why dcast() is fast | 1. Block-path detection (Phase 0a–0c) | 2. 64-bit packed keys and 96-bit fingerprint fallback | 3. Radix sort for shuffle-friendly input | 4. Permutation shortcut | Where the gap is narrowest | Cross-engine consistency | melt consistency | dcast consistency | Round-trip example | Headline numbers | Ubuntu 25.10 | Windows 11 Pro for Workstations | Session info
dataprep: performance and cross-engine consistency14 days ago
Overview | Mean vs median | Test environment | How to read these numbers | Cleaning pipeline (dataprep 0.1.5 → 0.1.8) | melt() — wide to long | Vary rows, 1 id + 9 value columns | Vary rows, 10 id (5 int + 5 chr) | Vary value columns, 1e3 rows, 1 id | Vary value columns, 1e3 rows, 10 id | melt() on Windows 11 Pro for Workstations | dcast() — long to wide | Vary n_long, 1 id, 10 levels | Vary levels, 1 id, 1e6 rows | Vary n_long, 1 id, 100 levels | Vary n_id, 1e6 rows, 10 levels | dcast() on Windows 11 Pro for Workstations | Summary of speedups | Ubuntu 25.10 | Windows 11 Pro for Workstations | Cross-engine consistency | melt consistency | dcast consistency | Reproducing the benchmarks | Session info
dataprep: upgrading from 0.1.5 to 0.1.814 days ago
What changed | Interface changes | Minimal reproduction of the boundary change | Minimal reproduction of the multi-column change | Implementation evolution | Quantified effect on a full-year dataset | Where the difference comes from | Migration checklist | What did not change | Performance reference | Test environments | Where to go next | Session info
dataprep: a leakage-free preprocessing workflow14 days ago
The data-leakage problem | A minimal example | What is stored in the plan | A note on degenerate columns | Adding or removing steps | Column-order independence | Missing-column detection | Workflow with dataprep() | Reporting | Full applied workflow | When NOT to preprocess | Test environment | Where to go next | Session info
dataprep: cleaning pipeline14 days ago
Quick start | Overview | Why obsedele changed in 0.1.8 | Boundary behaviour | condextr vs percoutl | Why obsedele is re-applied after condextr | Step-by-step walkthrough | Step 1 — Variable deletion | Step 2 — Observation deletion | Before cleaning | Step 3 — Conditional extremum outlier removal | After cleaning | Step 4 — Short-period interpolation | One-call pipeline: dataprep() | Inspecting the plan without running it | Missing-value diagnostics | Additional cleaning helpers | When NOT to preprocess | Where to go next | Session info
dataprep: descriptive statistics and diagnostic plots14 days ago
Overview | Descriptive statistics | Line plot, numeric variable names | Selected statistics | Bar chart, character variable names | Control facet layout | Full-size data | The underlying table | Percentile plots | Full percentile range | Top percentiles only | Bottom percentiles only | Percentile curves of the raw data | Numeric axis control | Combining with ggplot2 | A diagnostic workflow | Where to go next | Session info
dataprep: design philosophy and preprocessing methodology14 days ago
The problem this package solves | The four-step pipeline | Variable deletion: three principles | 1. Signal-to-noise ratio as the primary threshold | 2. Physical relevance | 3. Missing-run length, not just missing fraction | Observation deletion: two principles | 1. Detect the pattern, don't filter the points | 2. The half-hour window is the physical mixing time | How it is implemented: anchor-based scan | Implementation evolution | Why "both sides" and not "one side"? | Subsets: group and period | Re-applied after outlier deletion | Outlier removal: two principles | 1. Per-point decisions, not one-shot thresholds | 2. Outlier deletion inherits the gap-length constraint | Missing-value interpolation: two principles | 1. Vertical, not horizontal | 2. Segment the series before interpolating | Why this order, and not any other | Evaluation metrics | Number of values deleted | Sample retention | Residual outliers | Newly created outliers | Uncertainty analysis | The package beyond the four steps | Related documentation | Funding | Session info
dataprep: fast reshaping with melt() and dcast()14 days ago
Why another reshape implementation | Test environment | Wide to long with melt() | Basic usage | Measure-side specification | Automatic ID inference | Custom column names and na.rm | Memory layout: major = "row" vs major = "col" | Thread control | Why melt() is fast | 1. Two layout paths, selected by major | 2. SIMD streaming stores | 3. Hugepage hint for large outputs | 4. Fast paths for small inputs | 5. Cache-aware block sizing | 6. Per-call caches for SEXP and factor levels | Long to wide with dcast() | Formula interface | Fill missing cells | Aggregating duplicate pairs | na.rm | Why dcast() is fast | 1. Block-path detection (Phase 0a–0c) | 2. 64-bit packed keys and 96-bit fingerprint fallback | 3. Radix sort for shuffle-friendly input | 4. Permutation shortcut | Where the gap is narrowest | Cross-engine consistency | melt consistency | dcast consistency | Round-trip example | Headline numbers | Ubuntu 25.10 | Windows 11 Pro for Workstations | Session info
dataprep: performance and cross-engine consistency14 days ago
Overview | Mean vs median | Test environment | How to read these numbers | Cleaning pipeline (dataprep 0.1.5 → 0.1.8) | melt() — wide to long | Vary rows, 1 id + 9 value columns | Vary rows, 10 id (5 int + 5 chr) | Vary value columns, 1e3 rows, 1 id | Vary value columns, 1e3 rows, 10 id | melt() on Windows 11 Pro for Workstations | dcast() — long to wide | Vary n_long, 1 id, 10 levels | Vary levels, 1 id, 1e6 rows | Vary n_long, 1 id, 100 levels | Vary n_id, 1e6 rows, 10 levels | dcast() on Windows 11 Pro for Workstations | Summary of speedups | Ubuntu 25.10 | Windows 11 Pro for Workstations | Cross-engine consistency | melt consistency | dcast consistency | Reproducing the benchmarks | Session info
dataprep: upgrading from 0.1.5 to 0.1.814 days ago
What changed | Interface changes | Minimal reproduction of the boundary change | Minimal reproduction of the multi-column change | Implementation evolution | Quantified effect on a full-year dataset | Where the difference comes from | Migration checklist | What did not change | Performance reference | Test environments | Where to go next | Session info