diff --git a/.agents/skills/data-visualization/SKILL.md b/.agents/skills/data-visualization/SKILL.md deleted file mode 100644 index 409cce60..00000000 --- a/.agents/skills/data-visualization/SKILL.md +++ /dev/null @@ -1,305 +0,0 @@ ---- -name: data-visualization -description: Create effective data visualizations with Python (matplotlib, seaborn, plotly). Use when building charts, choosing the right chart type for a dataset, creating publication-quality figures, or applying design principles like accessibility and color theory. -user-invocable: false ---- - -# Data Visualization Skill - -Chart selection guidance, Python visualization code patterns, design principles, and accessibility considerations for creating effective data visualizations. - -## Chart Selection Guide - -### Choose by Data Relationship - -| What You're Showing | Best Chart | Alternatives | -|---|---|---| -| **Trend over time** | Line chart | Area chart (if showing cumulative or composition) | -| **Comparison across categories** | Vertical bar chart | Horizontal bar (many categories), lollipop chart | -| **Ranking** | Horizontal bar chart | Dot plot, slope chart (comparing two periods) | -| **Part-to-whole composition** | Stacked bar chart | Treemap (hierarchical), waffle chart | -| **Composition over time** | Stacked area chart | 100% stacked bar (for proportion focus) | -| **Distribution** | Histogram | Box plot (comparing groups), violin plot, strip plot | -| **Correlation (2 variables)** | Scatter plot | Bubble chart (add 3rd variable as size) | -| **Correlation (many variables)** | Heatmap (correlation matrix) | Pair plot | -| **Geographic patterns** | Choropleth map | Bubble map, hex map | -| **Flow / process** | Sankey diagram | Funnel chart (sequential stages) | -| **Relationship network** | Network graph | Chord diagram | -| **Performance vs. target** | Bullet chart | Gauge (single KPI only) | -| **Multiple KPIs at once** | Small multiples | Dashboard with separate charts | - -### When NOT to Use Certain Charts - -- **Pie charts**: Avoid unless <6 categories and exact proportions matter less than rough comparison. Humans are bad at comparing angles. Use bar charts instead. -- **3D charts**: Never. They distort perception and add no information. -- **Dual-axis charts**: Use cautiously. They can mislead by implying correlation. Clearly label both axes if used. -- **Stacked bar (many categories)**: Hard to compare middle segments. Use small multiples or grouped bars instead. -- **Donut charts**: Slightly better than pie charts but same fundamental issues. Use for single KPI display at most. - -## Python Visualization Code Patterns - -### Setup and Style - -```python -import matplotlib.pyplot as plt -import matplotlib.ticker as mticker -import seaborn as sns -import pandas as pd -import numpy as np - -# Professional style setup -plt.style.use('seaborn-v0_8-whitegrid') -plt.rcParams.update({ - 'figure.figsize': (10, 6), - 'figure.dpi': 150, - 'font.size': 11, - 'axes.titlesize': 14, - 'axes.titleweight': 'bold', - 'axes.labelsize': 11, - 'xtick.labelsize': 10, - 'ytick.labelsize': 10, - 'legend.fontsize': 10, - 'figure.titlesize': 16, -}) - -# Colorblind-friendly palettes -PALETTE_CATEGORICAL = ['#4C72B0', '#DD8452', '#55A868', '#C44E52', '#8172B3', '#937860'] -PALETTE_SEQUENTIAL = 'YlOrRd' -PALETTE_DIVERGING = 'RdBu_r' -``` - -### Line Chart (Time Series) - -```python -fig, ax = plt.subplots(figsize=(10, 6)) - -for label, group in df.groupby('category'): - ax.plot(group['date'], group['value'], label=label, linewidth=2) - -ax.set_title('Metric Trend by Category', fontweight='bold') -ax.set_xlabel('Date') -ax.set_ylabel('Value') -ax.legend(loc='upper left', frameon=True) -ax.spines['top'].set_visible(False) -ax.spines['right'].set_visible(False) - -# Format dates on x-axis -fig.autofmt_xdate() - -plt.tight_layout() -plt.savefig('trend_chart.png', dpi=150, bbox_inches='tight') -``` - -### Bar Chart (Comparison) - -```python -fig, ax = plt.subplots(figsize=(10, 6)) - -# Sort by value for easy reading -df_sorted = df.sort_values('metric', ascending=True) - -bars = ax.barh(df_sorted['category'], df_sorted['metric'], color=PALETTE_CATEGORICAL[0]) - -# Add value labels -for bar in bars: - width = bar.get_width() - ax.text(width + 0.5, bar.get_y() + bar.get_height()/2, - f'{width:,.0f}', ha='left', va='center', fontsize=10) - -ax.set_title('Metric by Category (Ranked)', fontweight='bold') -ax.set_xlabel('Metric Value') -ax.spines['top'].set_visible(False) -ax.spines['right'].set_visible(False) - -plt.tight_layout() -plt.savefig('bar_chart.png', dpi=150, bbox_inches='tight') -``` - -### Histogram (Distribution) - -```python -fig, ax = plt.subplots(figsize=(10, 6)) - -ax.hist(df['value'], bins=30, color=PALETTE_CATEGORICAL[0], edgecolor='white', alpha=0.8) - -# Add mean and median lines -mean_val = df['value'].mean() -median_val = df['value'].median() -ax.axvline(mean_val, color='red', linestyle='--', linewidth=1.5, label=f'Mean: {mean_val:,.1f}') -ax.axvline(median_val, color='green', linestyle='--', linewidth=1.5, label=f'Median: {median_val:,.1f}') - -ax.set_title('Distribution of Values', fontweight='bold') -ax.set_xlabel('Value') -ax.set_ylabel('Frequency') -ax.legend() -ax.spines['top'].set_visible(False) -ax.spines['right'].set_visible(False) - -plt.tight_layout() -plt.savefig('histogram.png', dpi=150, bbox_inches='tight') -``` - -### Heatmap - -```python -fig, ax = plt.subplots(figsize=(10, 8)) - -# Pivot data for heatmap format -pivot = df.pivot_table(index='row_dim', columns='col_dim', values='metric', aggfunc='sum') - -sns.heatmap(pivot, annot=True, fmt=',.0f', cmap='YlOrRd', - linewidths=0.5, ax=ax, cbar_kws={'label': 'Metric Value'}) - -ax.set_title('Metric by Row Dimension and Column Dimension', fontweight='bold') -ax.set_xlabel('Column Dimension') -ax.set_ylabel('Row Dimension') - -plt.tight_layout() -plt.savefig('heatmap.png', dpi=150, bbox_inches='tight') -``` - -### Small Multiples - -```python -categories = df['category'].unique() -n_cats = len(categories) -n_cols = min(3, n_cats) -n_rows = (n_cats + n_cols - 1) // n_cols - -fig, axes = plt.subplots(n_rows, n_cols, figsize=(5*n_cols, 4*n_rows), sharex=True, sharey=True) -axes = axes.flatten() if n_cats > 1 else [axes] - -for i, cat in enumerate(categories): - ax = axes[i] - subset = df[df['category'] == cat] - ax.plot(subset['date'], subset['value'], color=PALETTE_CATEGORICAL[i % len(PALETTE_CATEGORICAL)]) - ax.set_title(cat, fontsize=12) - ax.spines['top'].set_visible(False) - ax.spines['right'].set_visible(False) - -# Hide empty subplots -for j in range(i+1, len(axes)): - axes[j].set_visible(False) - -fig.suptitle('Trends by Category', fontsize=14, fontweight='bold', y=1.02) -plt.tight_layout() -plt.savefig('small_multiples.png', dpi=150, bbox_inches='tight') -``` - -### Number Formatting Helpers - -```python -def format_number(val, format_type='number'): - """Format numbers for chart labels.""" - if format_type == 'currency': - if abs(val) >= 1e9: - return f'${val/1e9:.1f}B' - elif abs(val) >= 1e6: - return f'${val/1e6:.1f}M' - elif abs(val) >= 1e3: - return f'${val/1e3:.1f}K' - else: - return f'${val:,.0f}' - elif format_type == 'percent': - return f'{val:.1f}%' - elif format_type == 'number': - if abs(val) >= 1e9: - return f'{val/1e9:.1f}B' - elif abs(val) >= 1e6: - return f'{val/1e6:.1f}M' - elif abs(val) >= 1e3: - return f'{val/1e3:.1f}K' - else: - return f'{val:,.0f}' - return str(val) - -# Usage with axis formatter -ax.yaxis.set_major_formatter(mticker.FuncFormatter(lambda x, p: format_number(x, 'currency'))) -``` - -### Interactive Charts with Plotly - -```python -import plotly.express as px -import plotly.graph_objects as go - -# Simple interactive line chart -fig = px.line(df, x='date', y='value', color='category', - title='Interactive Metric Trend', - labels={'value': 'Metric Value', 'date': 'Date'}) -fig.update_layout(hovermode='x unified') -fig.write_html('interactive_chart.html') -fig.show() - -# Interactive scatter with hover data -fig = px.scatter(df, x='metric_a', y='metric_b', color='category', - size='size_metric', hover_data=['name', 'detail_field'], - title='Correlation Analysis') -fig.show() -``` - -## Design Principles - -### Color - -- **Use color purposefully**: Color should encode data, not decorate -- **Highlight the story**: Use a bright accent color for the key insight; grey everything else -- **Sequential data**: Use a single-hue gradient (light to dark) for ordered values -- **Diverging data**: Use a two-hue gradient with neutral midpoint for data with a meaningful center -- **Categorical data**: Use distinct hues, maximum 6-8 before it gets confusing -- **Avoid red/green only**: 8% of men are red-green colorblind. Use blue/orange as primary pair - -### Typography - -- **Title states the insight**: "Revenue grew 23% YoY" beats "Revenue by Month" -- **Subtitle adds context**: Date range, filters applied, data source -- **Axis labels are readable**: Never rotated 90 degrees if avoidable. Shorten or wrap instead -- **Data labels add precision**: Use on key points, not every single bar -- **Annotation highlights**: Call out specific points with text annotations - -### Layout - -- **Reduce chart junk**: Remove gridlines, borders, backgrounds that don't carry information -- **Sort meaningfully**: Categories sorted by value (not alphabetically) unless there's a natural order (months, stages) -- **Appropriate aspect ratio**: Time series wider than tall (3:1 to 2:1); comparisons can be squarer -- **White space is good**: Don't cram charts together. Give each visualization room to breathe - -### Accuracy - -- **Bar charts start at zero**: Always. A bar from 95 to 100 exaggerates a 5% difference -- **Line charts can have non-zero baselines**: When the range of variation is meaningful -- **Consistent scales across panels**: When comparing multiple charts, use the same axis range -- **Show uncertainty**: Error bars, confidence intervals, or ranges when data is uncertain -- **Label your axes**: Never make the reader guess what the numbers mean - -## Accessibility Considerations - -### Color Blindness - -- Never rely on color alone to distinguish data series -- Add pattern fills, different line styles (solid, dashed, dotted), or direct labels -- Test with a colorblind simulator (e.g., Coblis, Sim Daltonism) -- Use the colorblind-friendly palette: `sns.color_palette("colorblind")` - -### Screen Readers - -- Include alt text describing the chart's key finding -- Provide a data table alternative alongside the visualization -- Use semantic titles and labels - -### General Accessibility - -- Sufficient contrast between data elements and background -- Text size minimum 10pt for labels, 12pt for titles -- Avoid conveying information only through spatial position (add labels) -- Consider printing: does the chart work in black and white? - -### Accessibility Checklist - -Before sharing a visualization: -- [ ] Chart works without color (patterns, labels, or line styles differentiate series) -- [ ] Text is readable at standard zoom level -- [ ] Title describes the insight, not just the data -- [ ] Axes are labeled with units -- [ ] Legend is clear and positioned without obscuring data -- [ ] Data source and date range are noted diff --git a/.agents/skills/dataverse-python-advanced-patterns/SKILL.md b/.agents/skills/dataverse-python-advanced-patterns/SKILL.md deleted file mode 100644 index 921ab603..00000000 --- a/.agents/skills/dataverse-python-advanced-patterns/SKILL.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -name: dataverse-python-advanced-patterns -description: 'Generate production code for Dataverse SDK using advanced patterns, error handling, and optimization techniques.' ---- - -You are a Dataverse SDK for Python expert. Generate production-ready Python code that demonstrates: - -1. **Error handling & retry logic** — Catch DataverseError, check is_transient, implement exponential backoff. -2. **Batch operations** — Bulk create/update/delete with proper error recovery. -3. **OData query optimization** — Filter, select, orderby, expand, and paging with correct logical names. -4. **Table metadata** — Create/inspect/delete custom tables with proper column type definitions (IntEnum for option sets). -5. **Configuration & timeouts** — Use DataverseConfig for http_retries, http_backoff, http_timeout, language_code. -6. **Cache management** — Flush picklist cache when metadata changes. -7. **File operations** — Upload large files in chunks; handle chunked vs. simple upload. -8. **Pandas integration** — Use PandasODataClient for DataFrame workflows when appropriate. - -Include docstrings, type hints, and link to official API reference for each class/method used. diff --git a/.agents/skills/python-performance-optimization/SKILL.md b/.agents/skills/python-performance-optimization/SKILL.md deleted file mode 100644 index 31b820c0..00000000 --- a/.agents/skills/python-performance-optimization/SKILL.md +++ /dev/null @@ -1,851 +0,0 @@ ---- -name: python-performance-optimization -description: Profile and optimize Python code using cProfile, memory profilers, and performance best practices. Use when debugging slow Python code, optimizing bottlenecks, or improving application performance. ---- - -# Python Performance Optimization - -Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices. - -## When to Use This Skill - -- Identifying performance bottlenecks in Python applications -- Reducing application latency and response times -- Optimizing CPU-intensive operations -- Reducing memory consumption and memory leaks -- Improving database query performance -- Optimizing I/O operations -- Speeding up data processing pipelines -- Implementing high-performance algorithms -- Profiling production applications - -## Core Concepts - -### 1. Profiling Types - -- **CPU Profiling**: Identify time-consuming functions -- **Memory Profiling**: Track memory allocation and leaks -- **Line Profiling**: Profile at line-by-line granularity -- **Call Graph**: Visualize function call relationships - -### 2. Performance Metrics - -- **Execution Time**: How long operations take -- **Memory Usage**: Peak and average memory consumption -- **CPU Utilization**: Processor usage patterns -- **I/O Wait**: Time spent on I/O operations - -### 3. Optimization Strategies - -- **Algorithmic**: Better algorithms and data structures -- **Implementation**: More efficient code patterns -- **Parallelization**: Multi-threading/processing -- **Caching**: Avoid redundant computation -- **Native Extensions**: C/Rust for critical paths - -## Quick Start - -### Basic Timing - -```python -import time - -def measure_time(): - """Simple timing measurement.""" - start = time.time() - - # Your code here - result = sum(range(1000000)) - - elapsed = time.time() - start - print(f"Execution time: {elapsed:.4f} seconds") - return result - -# Better: use timeit for accurate measurements -import timeit - -execution_time = timeit.timeit( - "sum(range(1000000))", - number=100 -) -print(f"Average time: {execution_time/100:.6f} seconds") -``` - -## Profiling Tools - -### Pattern 1: cProfile - CPU Profiling - -```python -import cProfile -import pstats -from pstats import SortKey - -def slow_function(): - """Function to profile.""" - total = 0 - for i in range(1000000): - total += i - return total - -def another_function(): - """Another function.""" - return [i**2 for i in range(100000)] - -def main(): - """Main function to profile.""" - result1 = slow_function() - result2 = another_function() - return result1, result2 - -# Profile the code -if __name__ == "__main__": - profiler = cProfile.Profile() - profiler.enable() - - main() - - profiler.disable() - - # Print stats - stats = pstats.Stats(profiler) - stats.sort_stats(SortKey.CUMULATIVE) - stats.print_stats(10) # Top 10 functions - - # Save to file for later analysis - stats.dump_stats("profile_output.prof") -``` - -**Command-line profiling:** - -```bash -# Profile a script -python -m cProfile -o output.prof script.py - -# View results -python -m pstats output.prof -# In pstats: -# sort cumtime -# stats 10 -``` - -### Pattern 2: line_profiler - Line-by-Line Profiling - -```python -# Install: pip install line-profiler - -# Add @profile decorator (line_profiler provides this) -@profile -def process_data(data): - """Process data with line profiling.""" - result = [] - for item in data: - processed = item * 2 - result.append(processed) - return result - -# Run with: -# kernprof -l -v script.py -``` - -**Manual line profiling:** - -```python -from line_profiler import LineProfiler - -def process_data(data): - """Function to profile.""" - result = [] - for item in data: - processed = item * 2 - result.append(processed) - return result - -if __name__ == "__main__": - lp = LineProfiler() - lp.add_function(process_data) - - data = list(range(100000)) - - lp_wrapper = lp(process_data) - lp_wrapper(data) - - lp.print_stats() -``` - -### Pattern 3: memory_profiler - Memory Usage - -```python -# Install: pip install memory-profiler - -from memory_profiler import profile - -@profile -def memory_intensive(): - """Function that uses lots of memory.""" - # Create large list - big_list = [i for i in range(1000000)] - - # Create large dict - big_dict = {i: i**2 for i in range(100000)} - - # Process data - result = sum(big_list) - - return result - -if __name__ == "__main__": - memory_intensive() - -# Run with: -# python -m memory_profiler script.py -``` - -### Pattern 4: py-spy - Production Profiling - -```bash -# Install: pip install py-spy - -# Profile a running Python process -py-spy top --pid 12345 - -# Generate flamegraph -py-spy record -o profile.svg --pid 12345 - -# Profile a script -py-spy record -o profile.svg -- python script.py - -# Dump current call stack -py-spy dump --pid 12345 -``` - -## Optimization Patterns - -### Pattern 5: List Comprehensions vs Loops - -```python -import timeit - -# Slow: Traditional loop -def slow_squares(n): - """Create list of squares using loop.""" - result = [] - for i in range(n): - result.append(i**2) - return result - -# Fast: List comprehension -def fast_squares(n): - """Create list of squares using comprehension.""" - return [i**2 for i in range(n)] - -# Benchmark -n = 100000 - -slow_time = timeit.timeit(lambda: slow_squares(n), number=100) -fast_time = timeit.timeit(lambda: fast_squares(n), number=100) - -print(f"Loop: {slow_time:.4f}s") -print(f"Comprehension: {fast_time:.4f}s") -print(f"Speedup: {slow_time/fast_time:.2f}x") - -# Even faster for simple operations: map -def faster_squares(n): - """Use map for even better performance.""" - return list(map(lambda x: x**2, range(n))) -``` - -### Pattern 6: Generator Expressions for Memory - -```python -import sys - -def list_approach(): - """Memory-intensive list.""" - data = [i**2 for i in range(1000000)] - return sum(data) - -def generator_approach(): - """Memory-efficient generator.""" - data = (i**2 for i in range(1000000)) - return sum(data) - -# Memory comparison -list_data = [i for i in range(1000000)] -gen_data = (i for i in range(1000000)) - -print(f"List size: {sys.getsizeof(list_data)} bytes") -print(f"Generator size: {sys.getsizeof(gen_data)} bytes") - -# Generators use constant memory regardless of size -``` - -### Pattern 7: String Concatenation - -```python -import timeit - -def slow_concat(items): - """Slow string concatenation.""" - result = "" - for item in items: - result += str(item) - return result - -def fast_concat(items): - """Fast string concatenation with join.""" - return "".join(str(item) for item in items) - -def faster_concat(items): - """Even faster with list.""" - parts = [str(item) for item in items] - return "".join(parts) - -items = list(range(10000)) - -# Benchmark -slow = timeit.timeit(lambda: slow_concat(items), number=100) -fast = timeit.timeit(lambda: fast_concat(items), number=100) -faster = timeit.timeit(lambda: faster_concat(items), number=100) - -print(f"Concatenation (+): {slow:.4f}s") -print(f"Join (generator): {fast:.4f}s") -print(f"Join (list): {faster:.4f}s") -``` - -### Pattern 8: Dictionary Lookups vs List Searches - -```python -import timeit - -# Create test data -size = 10000 -items = list(range(size)) -lookup_dict = {i: i for i in range(size)} - -def list_search(items, target): - """O(n) search in list.""" - return target in items - -def dict_search(lookup_dict, target): - """O(1) search in dict.""" - return target in lookup_dict - -target = size - 1 # Worst case for list - -# Benchmark -list_time = timeit.timeit( - lambda: list_search(items, target), - number=1000 -) -dict_time = timeit.timeit( - lambda: dict_search(lookup_dict, target), - number=1000 -) - -print(f"List search: {list_time:.6f}s") -print(f"Dict search: {dict_time:.6f}s") -print(f"Speedup: {list_time/dict_time:.0f}x") -``` - -### Pattern 9: Local Variable Access - -```python -import timeit - -# Global variable (slow) -GLOBAL_VALUE = 100 - -def use_global(): - """Access global variable.""" - total = 0 - for i in range(10000): - total += GLOBAL_VALUE - return total - -def use_local(): - """Use local variable.""" - local_value = 100 - total = 0 - for i in range(10000): - total += local_value - return total - -# Local is faster -global_time = timeit.timeit(use_global, number=1000) -local_time = timeit.timeit(use_local, number=1000) - -print(f"Global access: {global_time:.4f}s") -print(f"Local access: {local_time:.4f}s") -print(f"Speedup: {global_time/local_time:.2f}x") -``` - -### Pattern 10: Function Call Overhead - -```python -import timeit - -def calculate_inline(): - """Inline calculation.""" - total = 0 - for i in range(10000): - total += i * 2 + 1 - return total - -def helper_function(x): - """Helper function.""" - return x * 2 + 1 - -def calculate_with_function(): - """Calculation with function calls.""" - total = 0 - for i in range(10000): - total += helper_function(i) - return total - -# Inline is faster due to no call overhead -inline_time = timeit.timeit(calculate_inline, number=1000) -function_time = timeit.timeit(calculate_with_function, number=1000) - -print(f"Inline: {inline_time:.4f}s") -print(f"Function calls: {function_time:.4f}s") -``` - -## Advanced Optimization - -### Pattern 11: NumPy for Numerical Operations - -```python -import timeit -import numpy as np - -def python_sum(n): - """Sum using pure Python.""" - return sum(range(n)) - -def numpy_sum(n): - """Sum using NumPy.""" - return np.arange(n).sum() - -n = 1000000 - -python_time = timeit.timeit(lambda: python_sum(n), number=100) -numpy_time = timeit.timeit(lambda: numpy_sum(n), number=100) - -print(f"Python: {python_time:.4f}s") -print(f"NumPy: {numpy_time:.4f}s") -print(f"Speedup: {python_time/numpy_time:.2f}x") - -# Vectorized operations -def python_multiply(): - """Element-wise multiplication in Python.""" - a = list(range(100000)) - b = list(range(100000)) - return [x * y for x, y in zip(a, b)] - -def numpy_multiply(): - """Vectorized multiplication in NumPy.""" - a = np.arange(100000) - b = np.arange(100000) - return a * b - -py_time = timeit.timeit(python_multiply, number=100) -np_time = timeit.timeit(numpy_multiply, number=100) - -print(f"\nPython multiply: {py_time:.4f}s") -print(f"NumPy multiply: {np_time:.4f}s") -print(f"Speedup: {py_time/np_time:.2f}x") -``` - -### Pattern 12: Caching with functools.lru_cache - -```python -from functools import lru_cache -import timeit - -def fibonacci_slow(n): - """Recursive fibonacci without caching.""" - if n < 2: - return n - return fibonacci_slow(n-1) + fibonacci_slow(n-2) - -@lru_cache(maxsize=None) -def fibonacci_fast(n): - """Recursive fibonacci with caching.""" - if n < 2: - return n - return fibonacci_fast(n-1) + fibonacci_fast(n-2) - -# Massive speedup for recursive algorithms -n = 30 - -slow_time = timeit.timeit(lambda: fibonacci_slow(n), number=1) -fast_time = timeit.timeit(lambda: fibonacci_fast(n), number=1000) - -print(f"Without cache (1 run): {slow_time:.4f}s") -print(f"With cache (1000 runs): {fast_time:.4f}s") - -# Cache info -print(f"Cache info: {fibonacci_fast.cache_info()}") -``` - -### Pattern 13: Using **slots** for Memory - -```python -import sys - -class RegularClass: - """Regular class with __dict__.""" - def __init__(self, x, y, z): - self.x = x - self.y = y - self.z = z - -class SlottedClass: - """Class with __slots__ for memory efficiency.""" - __slots__ = ['x', 'y', 'z'] - - def __init__(self, x, y, z): - self.x = x - self.y = y - self.z = z - -# Memory comparison -regular = RegularClass(1, 2, 3) -slotted = SlottedClass(1, 2, 3) - -print(f"Regular class size: {sys.getsizeof(regular)} bytes") -print(f"Slotted class size: {sys.getsizeof(slotted)} bytes") - -# Significant savings with many instances -regular_objects = [RegularClass(i, i+1, i+2) for i in range(10000)] -slotted_objects = [SlottedClass(i, i+1, i+2) for i in range(10000)] - -print(f"\nMemory for 10000 regular objects: ~{sys.getsizeof(regular) * 10000} bytes") -print(f"Memory for 10000 slotted objects: ~{sys.getsizeof(slotted) * 10000} bytes") -``` - -### Pattern 14: Multiprocessing for CPU-Bound Tasks - -```python -import multiprocessing as mp -import time - -def cpu_intensive_task(n): - """CPU-intensive calculation.""" - return sum(i**2 for i in range(n)) - -def sequential_processing(): - """Process tasks sequentially.""" - start = time.time() - results = [cpu_intensive_task(1000000) for _ in range(4)] - elapsed = time.time() - start - return elapsed, results - -def parallel_processing(): - """Process tasks in parallel.""" - start = time.time() - with mp.Pool(processes=4) as pool: - results = pool.map(cpu_intensive_task, [1000000] * 4) - elapsed = time.time() - start - return elapsed, results - -if __name__ == "__main__": - seq_time, seq_results = sequential_processing() - par_time, par_results = parallel_processing() - - print(f"Sequential: {seq_time:.2f}s") - print(f"Parallel: {par_time:.2f}s") - print(f"Speedup: {seq_time/par_time:.2f}x") -``` - -### Pattern 15: Async I/O for I/O-Bound Tasks - -```python -import asyncio -import aiohttp -import time -import requests - -urls = [ - "https://httpbin.org/delay/1", - "https://httpbin.org/delay/1", - "https://httpbin.org/delay/1", - "https://httpbin.org/delay/1", -] - -def synchronous_requests(): - """Synchronous HTTP requests.""" - start = time.time() - results = [] - for url in urls: - response = requests.get(url) - results.append(response.status_code) - elapsed = time.time() - start - return elapsed, results - -async def async_fetch(session, url): - """Async HTTP request.""" - async with session.get(url) as response: - return response.status - -async def asynchronous_requests(): - """Asynchronous HTTP requests.""" - start = time.time() - async with aiohttp.ClientSession() as session: - tasks = [async_fetch(session, url) for url in urls] - results = await asyncio.gather(*tasks) - elapsed = time.time() - start - return elapsed, results - -# Async is much faster for I/O-bound work -sync_time, sync_results = synchronous_requests() -async_time, async_results = asyncio.run(asynchronous_requests()) - -print(f"Synchronous: {sync_time:.2f}s") -print(f"Asynchronous: {async_time:.2f}s") -print(f"Speedup: {sync_time/async_time:.2f}x") -``` - -## Database Optimization - -### Pattern 16: Batch Database Operations - -```python -import sqlite3 -import time - -def create_db(): - """Create test database.""" - conn = sqlite3.connect(":memory:") - conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)") - return conn - -def slow_inserts(conn, count): - """Insert records one at a time.""" - start = time.time() - cursor = conn.cursor() - for i in range(count): - cursor.execute("INSERT INTO users (name) VALUES (?)", (f"User {i}",)) - conn.commit() # Commit each insert - elapsed = time.time() - start - return elapsed - -def fast_inserts(conn, count): - """Batch insert with single commit.""" - start = time.time() - cursor = conn.cursor() - data = [(f"User {i}",) for i in range(count)] - cursor.executemany("INSERT INTO users (name) VALUES (?)", data) - conn.commit() # Single commit - elapsed = time.time() - start - return elapsed - -# Benchmark -conn1 = create_db() -slow_time = slow_inserts(conn1, 1000) - -conn2 = create_db() -fast_time = fast_inserts(conn2, 1000) - -print(f"Individual inserts: {slow_time:.4f}s") -print(f"Batch insert: {fast_time:.4f}s") -print(f"Speedup: {slow_time/fast_time:.2f}x") -``` - -### Pattern 17: Query Optimization - -```python -# Use indexes for frequently queried columns -""" --- Slow: No index -SELECT * FROM users WHERE email = 'user@example.com'; - --- Fast: With index -CREATE INDEX idx_users_email ON users(email); -SELECT * FROM users WHERE email = 'user@example.com'; -""" - -# Use query planning -import sqlite3 - -conn = sqlite3.connect("example.db") -cursor = conn.cursor() - -# Analyze query performance -cursor.execute("EXPLAIN QUERY PLAN SELECT * FROM users WHERE email = ?", ("test@example.com",)) -print(cursor.fetchall()) - -# Use SELECT only needed columns -# Slow: SELECT * -# Fast: SELECT id, name -``` - -## Memory Optimization - -### Pattern 18: Detecting Memory Leaks - -```python -import tracemalloc -import gc - -def memory_leak_example(): - """Example that leaks memory.""" - leaked_objects = [] - - for i in range(100000): - # Objects added but never removed - leaked_objects.append([i] * 100) - - # In real code, this would be an unintended reference - -def track_memory_usage(): - """Track memory allocations.""" - tracemalloc.start() - - # Take snapshot before - snapshot1 = tracemalloc.take_snapshot() - - # Run code - memory_leak_example() - - # Take snapshot after - snapshot2 = tracemalloc.take_snapshot() - - # Compare - top_stats = snapshot2.compare_to(snapshot1, 'lineno') - - print("Top 10 memory allocations:") - for stat in top_stats[:10]: - print(stat) - - tracemalloc.stop() - -# Monitor memory -track_memory_usage() - -# Force garbage collection -gc.collect() -``` - -### Pattern 19: Iterators vs Lists - -```python -import sys - -def process_file_list(filename): - """Load entire file into memory.""" - with open(filename) as f: - lines = f.readlines() # Loads all lines - return sum(1 for line in lines if line.strip()) - -def process_file_iterator(filename): - """Process file line by line.""" - with open(filename) as f: - return sum(1 for line in f if line.strip()) - -# Iterator uses constant memory -# List loads entire file into memory -``` - -### Pattern 20: Weakref for Caches - -```python -import weakref - -class CachedResource: - """Resource that can be garbage collected.""" - def __init__(self, data): - self.data = data - -# Regular cache prevents garbage collection -regular_cache = {} - -def get_resource_regular(key): - """Get resource from regular cache.""" - if key not in regular_cache: - regular_cache[key] = CachedResource(f"Data for {key}") - return regular_cache[key] - -# Weak reference cache allows garbage collection -weak_cache = weakref.WeakValueDictionary() - -def get_resource_weak(key): - """Get resource from weak cache.""" - resource = weak_cache.get(key) - if resource is None: - resource = CachedResource(f"Data for {key}") - weak_cache[key] = resource - return resource - -# When no strong references exist, objects can be GC'd -``` - -## Benchmarking Tools - -### Custom Benchmark Decorator - -```python -import time -from functools import wraps - -def benchmark(func): - """Decorator to benchmark function execution.""" - @wraps(func) - def wrapper(*args, **kwargs): - start = time.perf_counter() - result = func(*args, **kwargs) - elapsed = time.perf_counter() - start - print(f"{func.__name__} took {elapsed:.6f} seconds") - return result - return wrapper - -@benchmark -def slow_function(): - """Function to benchmark.""" - time.sleep(0.5) - return sum(range(1000000)) - -result = slow_function() -``` - -### Performance Testing with pytest-benchmark - -```python -# Install: pip install pytest-benchmark - -def test_list_comprehension(benchmark): - """Benchmark list comprehension.""" - result = benchmark(lambda: [i**2 for i in range(10000)]) - assert len(result) == 10000 - -def test_map_function(benchmark): - """Benchmark map function.""" - result = benchmark(lambda: list(map(lambda x: x**2, range(10000)))) - assert len(result) == 10000 - -# Run with: pytest test_performance.py --benchmark-compare -``` - -## Best Practices - -1. **Profile before optimizing** - Measure to find real bottlenecks -2. **Focus on hot paths** - Optimize code that runs most frequently -3. **Use appropriate data structures** - Dict for lookups, set for membership -4. **Avoid premature optimization** - Clarity first, then optimize -5. **Use built-in functions** - They're implemented in C -6. **Cache expensive computations** - Use lru_cache -7. **Batch I/O operations** - Reduce system calls -8. **Use generators** for large datasets -9. **Consider NumPy** for numerical operations -10. **Profile production code** - Use py-spy for live systems - -## Common Pitfalls - -- Optimizing without profiling -- Using global variables unnecessarily -- Not using appropriate data structures -- Creating unnecessary copies of data -- Not using connection pooling for databases -- Ignoring algorithmic complexity -- Over-optimizing rare code paths -- Not considering memory usage diff --git a/.agents/skills/seo-audit/SKILL.md b/.agents/skills/seo-audit/SKILL.md deleted file mode 100644 index 1dbe4de9..00000000 --- a/.agents/skills/seo-audit/SKILL.md +++ /dev/null @@ -1,412 +0,0 @@ ---- -name: seo-audit -description: When the user wants to audit, review, or diagnose SEO issues on their site. Also use when the user mentions "SEO audit," "technical SEO," "why am I not ranking," "SEO issues," "on-page SEO," "meta tags review," "SEO health check," "my traffic dropped," "lost rankings," "not showing up in Google," "site isn't ranking," "Google update hit me," "page speed," "core web vitals," "crawl errors," or "indexing issues." Use this even if the user just says something vague like "my SEO is bad" or "help with SEO" — start with an audit. For building pages at scale to target keywords, see programmatic-seo. For adding structured data, see schema-markup. For AI search optimization, see ai-seo. -metadata: - version: 1.1.0 ---- - -# SEO Audit - -You are an expert in search engine optimization. Your goal is to identify SEO issues and provide actionable recommendations to improve organic search performance. - -## Initial Assessment - -**Check for product marketing context first:** -If `.agents/product-marketing-context.md` exists (or `.claude/product-marketing-context.md` in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task. - -Before auditing, understand: - -1. **Site Context** - - What type of site? (SaaS, e-commerce, blog, etc.) - - What's the primary business goal for SEO? - - What keywords/topics are priorities? - -2. **Current State** - - Any known issues or concerns? - - Current organic traffic level? - - Recent changes or migrations? - -3. **Scope** - - Full site audit or specific pages? - - Technical + on-page, or one focus area? - - Access to Search Console / analytics? - ---- - -## Audit Framework - -### Schema Markup Detection Limitation - -**`web_fetch` and `curl` cannot reliably detect structured data / schema markup.** - -Many CMS plugins (AIOSEO, Yoast, RankMath) inject JSON-LD via client-side JavaScript — it won't appear in static HTML or `web_fetch` output (which strips `