Skip to main content

Baselinr Dashboard v2.0

Internal dashboard MVP for Baselinr data profiling and drift detection. This dashboard visualizes profiling results, drift alerts, run history, and metrics across multi-warehouse environments.

🎯 Features​

Core Features​

  • Run History: View past profiling runs with filtering and search
  • Profiling Results: Detailed table and column-level metrics visualization
  • Drift Detection: Monitor data drift events with severity indicators
  • Metrics Dashboard: Aggregate KPIs and trends
  • Multi-Warehouse Support: PostgreSQL, Snowflake, MySQL, BigQuery, Redshift, SQLite
  • Export Functionality: Export data in JSON/CSV formats
  • AI Chat Assistant: Conversational interface for data quality investigation

Technical Stack​

Frontend:

  • Next.js 14 (App Router)
  • React 18
  • Tailwind CSS
  • Recharts for visualizations
  • TanStack Query for data fetching
  • Lucide React for icons

Backend:

  • FastAPI
  • SQLAlchemy
  • Pydantic
  • PostgreSQL

📁 Project Structure​

dashboard/
├── backend/                    # FastAPI backend
│   ├── main.py                # API endpoints
│   ├── models.py              # Pydantic models
│   ├── database.py            # Database client
│   ├── chat_models.py         # Chat API models
│   ├── chat_routes.py         # Chat API routes
│   ├── requirements.txt       # Python dependencies
│   └── sample_data_generator.py
├── frontend/                   # Next.js frontend
│   ├── app/                   # App router pages
│   │   ├── page.tsx           # Dashboard overview
│   │   ├── runs/              # Run history page
│   │   ├── drift/             # Drift alerts page
│   │   ├── tables/            # Table details page
│   │   ├── chat/              # AI Chat page
│   │   └── metrics/           # Metrics page
│   ├── components/            # Reusable components
│   │   ├── Sidebar.tsx
│   │   ├── KPICard.tsx
│   │   ├── RunsTable.tsx
│   │   ├── DriftAlertsTable.tsx
│   │   ├── FilterPanel.tsx
│   │   └── chat/             # Chat components
│   │       ├── ChatContainer.tsx
│   │       ├── ChatInput.tsx
│   │       └── ChatMessage.tsx
│   ├── types/                 # TypeScript types
│   │   ├── lineage.ts
│   │   └── chat.ts
│   ├── lib/                   # Utilities
│   │   └── api.ts             # API client
│   └── package.json
└── README.md                  # This file

🚀 Quick Start​

Prerequisites​

  • Node.js 18+ and npm/yarn
  • Python 3.10+
  • PostgreSQL database (Baselinr storage)
  • Existing Baselinr installation (Phase 1)

1. Backend Setup​

cd dashboard/backend

# Install dependencies
pip install -r requirements.txt

# Set environment variables (create .env file)
export BASELINR_DB_URL=postgresql://baselinr:baselinr@localhost:5433/baselinr
export API_HOST=0.0.0.0
export API_PORT=8000

# Generate sample data (optional)
python sample_data_generator.py

# Start the backend server
python main.py
# Or with uvicorn:
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Backend will be available at: http://localhost:8000

2. Frontend Setup​

cd dashboard/frontend

# Install dependencies
npm install
# or
yarn install

# Create .env.local file with:
# NEXT_PUBLIC_API_URL=http://localhost:8000

# Start the development server
npm run dev
# or
yarn dev

Frontend will be available at: http://localhost:3000

🔌 API Endpoints​

Dashboard Metrics​

  • GET /api/dashboard/metrics?warehouse=&days=30 - Get aggregate metrics

Run History​

  • GET /api/runs?warehouse=&schema=&table=&status=&days=30 - List profiling runs
  • GET /api/runs/{run_id} - Get detailed run results

Drift Detection​

  • GET /api/drift?warehouse=&table=&severity=&days=30 - List drift alerts

Table Metrics​

  • GET /api/tables/{table_name}/metrics?schema=&warehouse= - Get table metrics

Warehouses​

  • GET /api/warehouses - List available warehouses

Export​

  • GET /api/export/runs?format=json&warehouse=&days=30 - Export runs
  • GET /api/export/drift?format=json&warehouse=&days=30 - Export drift

Chat (AI Assistant)​

  • GET /api/chat/config - Get chat configuration status
  • POST /api/chat/message - Send a message to the chat agent
  • GET /api/chat/history/{session_id} - Get chat history for a session
  • DELETE /api/chat/session/{session_id} - Clear a chat session
  • GET /api/chat/tools - List available chat tools
  • GET /api/chat/sessions - List active chat sessions

📊 Sample Data​

To populate the dashboard with sample data for testing:

cd dashboard/backend
python sample_data_generator.py

This generates:

  • 100 profiling runs across all warehouse types
  • Column-level metrics for each run
  • Drift events for ~30% of runs

🎨 Customization​

Theme Colors​

Modify tailwind.config.ts to customize colors:

colors: {
  primary: {
    500: '#0ea5e9',  // Main brand color
    // ...
  },
}

Adding New Pages​

  1. Create a new page in frontend/app/your-page/page.tsx
  2. Add navigation link in components/Sidebar.tsx
  3. Create API endpoint in backend/main.py if needed

🔗 Integration with Baselinr Phase 1​

The dashboard connects to the Baselinr storage database to read:

  • baselinr_runs: Run history and metadata
  • baselinr_results: Column-level metrics
  • baselinr_events: Drift detection events
  • baselinr_table_state: Incremental profiling metadata (snapshot IDs, last decisions)

Ensure your Baselinr Phase 1 installation has created these tables.

🐳 Docker Setup (Optional)​

TODO: Add Docker Compose configuration for easy deployment

📈 Roadmap / Future Enhancements​

  • Real-time updates via WebSockets
  • Advanced filtering and saved views
  • Custom dashboards per user
  • Alert notifications (email, Slack)
  • Figma-based design refinements
  • CSV export implementation
  • Pagination for large datasets
  • Dark mode support
  • User authentication

🤝 Contributing​

This is an internal MVP. For feature requests or bug reports, please contact the Baselinr team.

📝 Environment Variables​

Backend (.env)​

BASELINR_DB_URL=postgresql://user:password@host:port/database
API_HOST=0.0.0.0
API_PORT=8000
CORS_ORIGINS=http://localhost:3000

# Chat/AI Configuration (optional)
LLM_ENABLED=true
LLM_PROVIDER=openai  # or "anthropic"
LLM_MODEL=gpt-4o-mini  # or "claude-3-5-sonnet-20241022"
OPENAI_API_KEY=sk-your-api-key
# ANTHROPIC_API_KEY=sk-ant-your-api-key  # if using Anthropic
CHAT_MAX_ITERATIONS=5
CHAT_MAX_HISTORY=20
CHAT_TOOL_TIMEOUT=30

# Or use a config file
BASELINR_CONFIG=/path/to/config.yml

Frontend (.env.local)​

NEXT_PUBLIC_API_URL=http://localhost:8000
NODE_ENV=development

💬 Chat Feature​

The dashboard includes an AI-powered chat assistant for data quality investigation.

Enabling Chat​

  1. Set LLM_ENABLED=true in your environment
  2. Configure your LLM provider (OpenAI or Anthropic)
  3. Provide the appropriate API key

Chat Capabilities​

The chat assistant can:

  • Query recent profiling runs
  • Investigate drift events and anomalies
  • Get table profiles and column statistics
  • Compare runs and analyze trends
  • Explore data lineage relationships
  • Search across tables

Example Queries​

  • "What tables have been profiled recently?"
  • "Show me high severity drift events"
  • "Are there any anomalies I should investigate?"
  • "Compare the last two runs for the customers table"
  • "What's the trend for null rate in the email column?"
  • "What are the upstream sources for orders table?"

🛠️ Development​

Backend Development​

cd backend
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Frontend Development​

cd frontend
npm run dev

Visit:

📦 Production Build​

Frontend​

cd frontend
npm run build
npm start

Backend​

cd backend
uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4

🐛 Troubleshooting​

Connection Errors​

  • Ensure Baselinr database is running
  • Check BASELINR_DB_URL environment variable
  • Verify database tables exist (baselinr_runs, baselinr_results, baselinr_events)

No Data Showing​

  • Run the sample data generator: python sample_data_generator.py
  • Or run Baselinr profiling: baselinr profile --config config.yml

CORS Errors​

  • Check CORS_ORIGINS in backend includes frontend URL
  • Verify NEXT_PUBLIC_API_URL in frontend points to backend

📄 License​

Internal use only - Baselinr Project