Initial commit: Korean stock value-investing AI pipeline

- 19개 마이크로서비스 (news-collector, score-engine, ta-engine, dart-collector,
  aux-signal, us-market, graph-engine, telegram-bot, dashboard-api, kis-api 등)
- 가치투자 스코어링 + 10공식 앙상블 보팅 (매직포뮬러/F-Score/Altman/PEG/
  모멘텀/Beneish/GP-A/G-Score/Amihud/BAB)
- 뉴스 수집→형태소→임베딩→중복제거→AI분석 파이프라인
- 기술적분석 + GAT 그래프신경망 + 미증시 동조 시그널

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyu
2026-05-20 21:33:56 +09:00
commit 6d3b0bacc0
60 changed files with 17818 additions and 0 deletions
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
{"id":"669c2fcf-b774-40a1-b874-21bbb5027474","name":"📈 AI 예측 성과 추적 (매일 16:10)","nodes":[{"id":"n1","name":"매일 16:10","type":"n8n-nodes-base.scheduleTrigger","typeVersion":1.1,"position":[100,300],"parameters":{"rule":{"interval":[{"field":"cronExpression","expression":"10 16 * * 1-5"}]}}},{"id":"n2","name":"추천종목 랭킹조회","type":"n8n-nodes-base.httpRequest","typeVersion":4.2,"position":[350,300],"parameters":{"method":"GET","url":"http://score-engine:8686/recommendations","options":{"timeout":30000}}},{"id":"n3","name":"오늘 성과 브리핑","type":"n8n-nodes-base.httpRequest","typeVersion":4.2,"position":[600,300],"parameters":{"method":"POST","url":"http://score-engine:8686/briefing/send","options":{"timeout":30000}}}],"connections":{"매일 16:10":{"main":[[{"node":"추천종목 랭킹조회","type":"main","index":0}]]},"추천종목 랭킹조회":{"main":[[{"node":"오늘 성과 브리핑","type":"main","index":0}]]}},"active":true}
+12
View File
@@ -0,0 +1,12 @@
*.log
logs/
data/
__pycache__/
.git/
node_modules/
*.csv
*.json
*.db
*.sqlite
volumes/
pg_backup/
+36
View File
@@ -0,0 +1,36 @@
# 비밀/환경
.env
.env.*
env-additions.txt
*.secret
secrets/
# DB 덤프/백업
pg_backup/
*.dump
*.sql.gz
*.tar.gz
# 마운트/모델/캐시
mnt/
models/
**/__pycache__/
*.pyc
*.pyo
.pytest_cache/
.ruff_cache/
.mypy_cache/
# IDE/OS
.DS_Store
.idea/
.vscode/
*.swp
*~
# Claude Code 로컬 설정
.claude/
# 로그
*.log
logs/
+577
View File
@@ -0,0 +1,577 @@
# Trading AI — Claude Code 프로젝트 가이드
> **역할**: 워렌 버핏 스타일 한국 주식 AI 투자 분석 전문가로 행동할 것.
> 가치투자 관점(ROE·영업이익률·부채비율·FCF)을 최우선으로 판단한다.
> 이 파일은 매 대화 시작 시 자동 로드됨 — 파일 탐색 없이 이 내용만으로 작업 시작.
---
## Claude 행동 원칙
### 1. 먼저 생각, 그 다음 코딩
- 가정은 명시적으로 밝힐 것. 불확실하면 질문.
- 해석이 여러 개면 나열하고 고를 것 — 혼자 결정 금지.
- 더 단순한 방법이 있으면 말할 것. 불필요한 복잡성에 반론.
### 2. 단순함 우선
- 요청한 것만 구현. 추측성 기능·추상화·유연성 추가 금지.
- "나중에 필요할 수도" 코드 금지. 200줄이 50줄로 가능하면 다시 짤 것.
- 불가능한 시나리오를 위한 에러 핸들링 금지.
### 3. 정밀한 변경
- 요청한 부분만 수정. 인접 코드 "개선" 금지.
- 기존 스타일 유지 (내 방식이 달라도).
- 내 변경으로 생긴 불필요한 import/변수/함수만 제거. 기존 dead code는 언급만.
### 4. 검증 기준 명시
- 다단계 작업 시 각 단계 완료 조건을 먼저 정의.
- Docker 서비스 변경 → 로그 확인 필수.
- DB 변경 → 쿼리로 확인 필수.
### 이 프로젝트 특이사항
- sudo 비밀번호 입력 불가 → sudo 필요 작업은 `! sudo <cmd>` 형태로 유저에게 요청.
- n8n 마이그레이션 충돌 이력 있음 → n8n 스키마 변경 시 반드시 백업 먼저.
- NAS fstab은 한 줄 유지 필수 (nfsvers=3).
- 상장폐지 필터(`is_active=true`) 빠지면 추천 결과 오염됨.
---
## 시스템 개요
뉴스 수집 → 형태소분석 → 임베딩 → 중복제거 → AI분석 → 재무스코어링 → 기술분석 → 매수/매도 추천 → 텔레그램 알림
---
## 인프라
| 항목 | 값 |
|------|-----|
| 서버 OS | Ubuntu 22.04 |
| GPU | GPU0: RTX 3060 12GB (EXAONE 추론) / GPU1: RTX 3070 8GB (bge-m3 임베딩) |
| NAS | `192.168.0.36:/volume1/trading → /mnt/nas` (nfsvers=3 필수) |
| DB 연결 | `postgres:5432`, DB명 `trading_ai`, user `kyu` |
| 환경변수 | `/home/kyu/trading/.env` |
| Docker | `docker compose` (파일: `/home/kyu/trading/docker-compose.yml`) |
| PostgreSQL 데이터 | `/mnt/nas/postgresql/data` (NAS 저장) |
---
## 서비스 목록
| 서비스 | 컨테이너 | 포트 | 내부IP | 역할 |
|--------|---------|------|--------|------|
| news-collector | trading-news-collector | 8787 | 172.30.0.16 | 뉴스수집+형태소+임베딩+Ollama분석 |
| bareunaapi | trading-bareunaapi | 5757 | 172.30.0.12 | 한국어 형태소분석 FastAPI 래퍼 |
| bareun | trading-bareun | 5656/9902 | 172.30.0.15 | 바른 NLP gRPC 서버 |
| dart-collector | trading-dart-collector | 8888 | 172.30.0.17 | DART 공시/재무제표 수집 |
| score-engine | trading-score-engine | 8686 | 172.30.0.19 | 종합 투자 스코어 계산+텔레그램 |
| ta-engine | trading-ta-engine | 8484 | 172.30.0.23 | 기술적분석 (MA/RSI/MACD/볼밴/스토캐스틱) |
| dashboard-api | trading-dashboard-api | 8989 | 172.30.0.22 | 대시보드 REST API |
| kis-api | trading-kis-api | 8585 | 172.30.0.18 | 네이버모바일 주가수집 + 매매시그널 |
| ollama | trading-ollama | 11434 | 172.30.0.13 | EXAONE 3.5 7.8B 추론(GPU0) + bge-m3 임베딩(GPU1) |
| n8n | trading-n8n | 5678 | 172.30.0.20 | 워크플로우 자동화 |
| n8n-worker | trading-n8n-worker | - | 172.30.0.21 | n8n Queue Worker |
| postgres | trading-postgres | 5432 | - | 메인 DB |
| redis | trading-redis | 6379 | 172.30.0.10 | 캐시/중복제거 |
| qdrant | trading-qdrant | 6333 | 172.30.0.11 | 뉴스 벡터 유사도 필터 |
| aux-signal | trading-aux-signal | 8282 | 172.30.0.25 | 보조 데이터: 네이버 컨센서스·기관/외국인 수급 + ECOS 매크로(USD/KRW, 국고채10년) |
| us-market | trading-us-market | 8383 | 172.30.0.24 | 미증시→한국 동조 시그널 (섹터ETF 14개 ±5점, 개별 페어 회귀 ±10점, 자동 페어발굴) |
| graph-engine | trading-graph-engine | 9090 | 172.30.0.27 | GAT 그래프 신경망 (12피처 노드 × 가격상관/섹터/뉴스공기 엣지) → graph_score |
| telegram-bot | trading-telegram-bot | - | 172.30.0.26 | 텔레그램 명령 처리 (/buy /sell /stock /deep /market) |
---
## 파일 구조 & 핵심 함수
### `/home/kyu/trading/news-collector/main.py` (21KB)
뉴스 수집 + AI 분석 파이프라인. 포트 8787.
| 함수/클래스 | 역할 |
|------------|------|
| `class S` | 전역 상태 (DB pool, Redis, Qdrant client) |
| `nhash(title, url)` | SHA256[:16] 뉴스 중복 해시 |
| `is_korean(text)` | 한글 포함 여부 체크 |
| `parse_rss_date(date_str)` | RSS pubDate → ISO timestamp |
| `@app.post("/collect/market")` | 네이버 금융 메인 뉴스 수집 |
| `@app.post("/collect/stocks")` | 종목별 네이버 뉴스 수집 |
| `@app.post("/collect/rss")` | 18개 RSS 피드 수집 |
| `@app.get("/sources")` | RSS 소스 목록 |
**파이프라인 순서** (함수 내부):
1. 바른API `/analyze` → 형태소분석 + 종목감지
2. Ollama `bge-m3` → 1024차원 임베딩
3. Qdrant → 코사인 유사도 ≥0.92 중복 제거
4. Ollama `exaone3.5:7.8b` (GPU0) → 호재/악재/중립 + intensity(1~5) + catalyst분류
5. PostgreSQL `news_analysis` 저장
**EXAONE 프롬프트**: 버핏 관점 (실적/수주/배당/리스크/모멘텀/기타 catalyst 분류)
**RSS 소스 (28개)**: 한국경제, 매일경제, 머니투데이, 이데일리, 연합뉴스, 조선비즈, 헤럴드경제, 아시아경제, 파이낸셜뉴스, SBS Biz, 뉴스1, 뉴시스 등 + 네이버금융 직접 크롤링
**스케줄러**:
- 평일 RSS: 8-18시 5분마다 (`rss_weekday`)
- 주말 RSS: 8-22시 15분마다 (`rss_weekend`) — 누적 학습용
- 평일 마켓: 9-17시 10분마다 (`market`, 네이버 금융 메인)
- raw 분석: 24시간 30분마다 200건 batch (`process_raw`) — 백로그 소화용으로 확장
- 주간 raw 백필: 일요일 02:00 전체 종목 × 50페이지 (`historical_raw_weekly`)
**수집/분석 분리** (대규모 백필용):
- `POST /collect/historical-raw?count=0&max_pages=100` — raw만 빠르게 수집 (LLM 스킵)
- `POST /process/raw?batch_size=200` — news_raw → news_analysis 점진 분석
- `GET /raw/stats` — 백필 진행률 (total/unprocessed/processed)
---
### `/home/kyu/trading/bareunaapi/main.py` (9.5KB)
한국어 형태소분석 FastAPI 래퍼. 포트 5757.
| 함수/클래스 | 역할 |
|------------|------|
| `class AppState` | bareun gRPC stub + Redis + 종목사전 |
| `class AnalyzeRequest / AnalyzeResponse` | 분석 요청/응답 모델 |
| `news_hash(title, url)` | Redis 중복제거용 해시 |
| `extract_morphemes(text)` | 바른API → 명사 추출 |
| `extract_stocks(text)` | 텍스트에서 종목코드 감지 |
| `build_filtered(nouns, stocks)` | 필터링된 텍스트 구성 |
| `scan_finance_terms(text)` | finance_dict.py 금융용어 매칭 |
| `_analyze(req)` | 핵심 분석 로직 |
| `@app.post("/analyze")` | 단건 분석 |
| `@app.post("/analyze/batch")` | 배치 분석 |
| `@app.get("/stocks")` | 로드된 종목 목록 |
| `@app.post("/stocks/refresh")` | KRX→Naver 종목 새로고침 |
| `@app.delete("/dedup/flush")` | Redis 중복제거 캐시 초기화 |
**관련 파일**:
- `bareunaapi/finance_dict.py` — 주식/금융 전문 용어 사전
- `bareunaapi/stock_loader.py` — KRX→Naver 폴백, ~3768 종목 로드
---
### `/home/kyu/trading/dart-collector/main.py` (34KB)
DART 공시/재무제표 수집. 포트 8888.
| 함수/클래스 | 역할 |
|------------|------|
| `class Stats` | 수집 통계 |
| `get_corp_code(stock_code)` | 종목코드 → DART corp_code 변환 |
| `get_corp_name(stock_code)` | 종목코드 → 기업명 |
| `calc_financial_ratios(key_items, prev_revenue)` | ROE/영업이익률/부채비율/FCF/매출성장률 계산 |
| `@app.post("/collect/disclosures")` | 공시 수집 |
| `@app.post("/collect/financials")` | 재무제표 수집 (300개 종목) |
| `@app.post("/collect/major")` | 주요 대형주 수집 |
| `@app.post("/collect/corps")` | DART 기업목록 갱신 |
| `@app.get("/corps")` | 기업목록 조회 |
| `@app.get("/corps/{stock_code}")` | 특정 기업 조회 |
| `@app.get("/financials/{stock_code}")` | 종목 재무데이터 |
| `@app.get("/stats")` | 수집 통계 |
---
### `/home/kyu/trading/score-engine/main.py` (27KB)
종합 투자 스코어 계산 + 텔레그램 알림. 포트 8686.
| 함수/클래스 | 역할 |
|------------|------|
| `get_recommendation(score)` | 점수→등급 변환 |
| `calc_fundamental_score(fin, per, pbr)` | 재무점수 산출 (-100~100) |
| `calc_magic_formula(fin, market_cap)` | 그린블라트 매직 포뮬러 (ROC + EY) 0~30점 |
| `calc_piotroski_score(curr, prev)` | 피오트로스키 F-Score 7신호 → -15~+15 |
| `calc_altman_z(fin, market_cap)` | 알트만 Z-Score 단순화 → 매수/매도/관망 신호 |
| `calc_peg(curr, prev, per)` | 린치 GARP — PER/이익성장률 → 신호 |
| `calc_momentum(conn, code)` | AQR 12-1개월 가격 모멘텀 → 신호 |
| `calc_beneish_simplified(curr, prev)` | Beneish M-Score 단순화 (분식 의심도) |
| `aggregate_signals(signals)` | 6공식 보팅 다수결 → (요약, 카운트) |
| `calc_trend_score(conn, code)` | 5년 ROE 일관성·추세 -30~+30 |
| `calc_dcf(fin, market_cap)` | 간이 DCF 내재가치 + 안전마진 |
| `calc_earnings_quality(fin)` | CFO/영업이익 비율 (분식의심 패널티) |
| `is_value_investable(fin, per, pbr, market_cap)` | 버핏 필터 통과 여부 |
| `@app.post("/score/calculate")` | 종목 점수 계산 실행 |
| `@app.post("/briefing/send")` | 텔레그램 브리핑 전송 |
| `@app.get("/ranking")` | 전체 종목 랭킹 |
| `@app.get("/recommendations")` | 추천 종목 목록 |
| `@app.get("/stock/{code}")` | 특정 종목 점수 상세 |
| `@app.get("/backtest")` | 추천 성과 백테스트 (수익률/승률/샤프/MDD/알파) |
| `@app.post("/learn-weights")` | 공식별 신호 → 7d 수익률 회귀로 가중치 학습 |
| `@app.get("/learn-weights")` | 현재 적용 중인 공식 가중치 조회 |
| `@app.post("/learn-pricing")` | D+E: 점수→30d 수익률 선형회귀 + Random Forest 학습 |
| `@app.get("/predict-price/{code}")` | 학습된 모델로 종목 30일 후 예상 수익률·가격 |
| `@app.get("/sector/concentration")` | 섹터 집중도 + 30% 초과 경고 |
**스코어링 공식 (실제 가중치)**:
```
종합점수 = 펀더멘털통합×0.24 + 뉴스(catalyst가중)×0.18 + 기술×0.15
+ DART공시×0.10 + 외국인수급×0.14 + 공매도×0.06
+ 가격모멘텀×0.03 + DCF안전마진×0.10
+ 시장레짐 보정(±5~10)
+ 앙상블 보팅(±18) ← 6공식 매수/매도 합 × 학습가중치
```
**10공식 앙상블 보팅** — 각 공식이 독립 신호 발신, 학습 가중치로 결합:
| 공식 | 출처/논문 | 데이터 | 매수 신호 |
|------|----------|--------|----------|
| 매직포뮬러 | Greenblatt | ROC + EY | magic_score ≥ 20 |
| F-Score | Piotroski 2000 | 7신호 (전년 대비) | f_score ≥ 6 |
| 알트만 Z | Altman 1968 | 6.72×ROA + 1.05×(시총/부채) | Z ≥ 2.6 |
| PEG | Lynch GARP | PER / 이익성장률 | PEG ≤ 1.5 |
| 12-1 모멘텀 | AQR (Carhart 1997) | (P_-21 / P_-252) - 1 | ≥ 10% |
| Beneish | Beneish 1999 | TATA·SGI·CFO/NI | 의심도 < 50 + CFO/NI > 1 |
| **GP/A** | Novy-Marx 2013 | 영업이익/총자산 (대체) | ≥ 15% |
| **G-Score** | Mohanram 2005 | 5신호 vs 섹터 중앙값 | ≥ 4 |
| **Amihud** | Amihud 2002 | avg(\|return\|/거래대금) | ≥ 100 (소형 알파) |
| **베타(BAB)** | Frazzini-Pedersen 2014 | 종목 vs KOSPI 60일 회귀 | β < 0.7 (저베타 알파) |
**학습 가중치** (`weight_config` 테이블):
- `POST /learn-weights?days=90` — 백테스트로 공식별 매수vs매도 그룹 7d 수익률 차이(edge) 측정
- edge가 큰 공식일수록 가중치 ↑ (정규화 후 합 = 6, 균등 시 각 1.0)
- 다음 점수 계산부터 자동 적용. 표본 부족 시 default(균등 1.0)
**펀더멘털통합** = `clip(buffett_score + trend_score + earnings_quality + magic_score + f_score_adj, -100, 100)`
- `buffett_score`: ROE / 영업이익률 / 부채비율 / 매출성장 / PER / PBR / FCF / 배당
- `trend_score`: 5년 ROE 일관성·추세 (-30~+30)
- `earnings_quality`: CFO/영업이익 ≥1 가산, <0.7 분식의심 패널티
- `magic_score`: ROC(영업이익/총자산) + EY(영업이익/EV) 임계값 합산 (0~30)
- `f_score_adj`: F-Score ≥6 +15, 5 +8, 4 +3, ≤2 -15 (가치함정 회피)
**매직 포뮬러 임계값** (한국 시장 보정):
- ROC: ≥25% +15 / ≥15% +10 / ≥8% +5
- EY: ≥15% +15 / ≥10% +10 / ≥6% +5
**피오트로스키 F-Score 7신호** (현재/전년 사업보고서 비교, 9중 2개는 데이터 부재로 생략):
1. ROA(NI/총자산) > 0 2. CFO > 0 3. ΔROA > 0 4. CFO > NI
5. Δ부채비율 < 0 6. Δ영업이익률 > 0 7. Δ자산회전율 > 0
**추천 등급** (점수 + 다수공식 동의 강제):
- 강력매수: 점수 ≥70 AND 매수보팅 ≥3
- 매수관심: 점수 ≥40 AND 매수보팅 ≥1 AND 매도보팅 <2
- 매도관심: 점수 ≤-30 OR 매도보팅 ≥3
- 강력매도: 점수 ≤-60 OR 매도보팅 ≥4
- 관망: 그 외
**버핏 가치투자 필터** (`is_value_investable`):
- `operating_profit > 0` 영업적자 제외
- ROE ≥ 10%
- 부채비율 ≤ 200%
- PER ≤ 60
- 시총 ≥ 100억
- `dart_corps.is_active=true` (상장폐지 제외)
---
### `/home/kyu/trading/ta-engine/main.py` (37KB)
기술적 분석 엔진. 포트 8484.
| 함수/클래스 | 역할 |
|------------|------|
| `_ema_series(values, period)` | EMA 계산 |
| `_ma(closes, n)` | 단순이동평균 |
| `_rsi(closes, period=14)` | RSI |
| `_macd(closes)` | MACD (12/26/9) |
| `_bollinger(closes, period=20)` | 볼린저밴드 |
| `_stochastic(highs, lows, closes, period=14)` | 스토캐스틱 K/D |
| `_vol_ratio(volumes, period=20)` | 거래량비율 |
| `calc_indicators(ohlcv)` | 전체 지표 계산 |
| `calc_tech_score(ind)` | 기술점수 (-100~100) |
| `calc_price_targets(price, ind, sig)` | 목표가/손절가 계산 |
| `analyze_position(price, buy_price, qty, ...)` | 포지션 분석 |
| `@app.get("/technical/{code}")` | 종목 기술분석 |
| `@app.get("/ranking")` | 기술점수 랭킹 |
| `@app.get("/buy-candidates")` | 매수 후보 |
| `@app.post("/analyze/all")` | 전체 종목 분석 |
| `@app.post("/analyze/{code}")` | 특정 종목 분석 |
| `@app.post("/position")` | 포지션 분석 |
| `@app.get("/report/{code}")` | 종합 리포트 |
**목표가 계산**:
- 진입가: `min(현재가, MA20×0.99)`
- T1: `max(볼밴상단, 현재가×1.05)`
- T2: `max((현재가+52주고가)/2, 현재가×1.10)`
- T3: `max(52주고가×0.97, 현재가×1.20)`
- 손절가: `min(MA60×0.98, 현재가×0.95)`
---
### `/home/kyu/trading/dashboard-api/main.py` (19KB)
대시보드 REST API. 포트 8989.
| 엔드포인트 | 역할 |
|-----------|------|
| `GET /api/summary` | 전체 요약 통계 |
| `GET /api/recent` | 최근 뉴스 분석 |
| `GET /api/ranking` | 종합 랭킹 |
| `GET /api/recommendations` | 추천 종목 |
| `GET /api/avoid` | 회피 종목 |
| `GET /api/signals` | 매매 시그널 |
| `GET /api/technical/{code}` | 기술분석 데이터 |
| `GET /api/buy-candidates` | 매수 후보 |
| `GET /api/alerts` | 알림 목록 |
| `GET /api/timeline` | 타임라인 |
| `GET /api/stock/{code}` | 종목 상세 |
| `GET /api/search` | 종목 검색 |
| `POST /api/position` | 포지션 등록 |
| `GET /api/report/{code}` | 종목 리포트 |
| `GET /api/fundamentals` | 전체 재무데이터 |
| `GET /api/fundamentals/{code}` | 종목 재무데이터 |
| `GET /api/name/{code}` | 종목명 조회 |
| `GET /` | 대시보드 HTML |
---
### `/home/kyu/trading/kis-api/main.py` (11KB)
네이버 모바일 주가수집 + 매매시그널. 포트 8585.
| 함수/엔드포인트 | 역할 |
|---------------|------|
| `class Stats` | 수집 통계 |
| `calc_signal(p, news_sc, dart_sc)` | 뉴스+공시점수 → 매매시그널 |
| `GET /price/{code}` | 단일 종목 주가 |
| `GET /prices` | 전체 주가 목록 |
| `GET /signals` | 전체 시그널 |
| `GET /signals/{code}` | 종목별 시그널 |
| `POST /collect` | 주가 수집 실행 |
**Redis DB 할당**:
- db=0: n8n Queue
- db=1: bareunaapi 중복제거
- db=3: kis-api/score-engine (`price:{code}`, `prices:last_update`)
- db=4: news-collector (`news:naver:{hash}`)
- db=5: ta-engine (`ta:{code}`)
---
### `/home/kyu/trading/aux-signal/main.py`
보조 데이터 수집. 포트 8282.
| 엔드포인트 | 역할 |
|-----------|------|
| `POST /collect/naver` | 네이버 integration API → 컨센서스 + 기관/외국인 일별 수급 |
| `POST /collect/macro` | 한국은행 ECOS → USD/KRW 환율, 국고채 10년 |
| `GET /consensus/{code}` | 종목 컨센서스 (목표가/투자의견) |
| `GET /flow/{code}` | 기관/외국인 일별 순매수 |
| `GET /macro/latest` | 최신 매크로 지표 |
| `GET /macro/{indicator}` | 지표별 시계열 |
스케줄: 영업일 18:00 네이버 수급/컨센서스, 08:30 ECOS 매크로.
---
### `/home/kyu/trading/us-market/main.py`
미국증시 동조 시그널. 포트 8383.
| 엔드포인트 | 역할 |
|-----------|------|
| `POST /collect` | Finnhub/AlphaVantage → 미국 ETF·개별주 가격 |
| `POST /collect/backfill` | 신규 페어용 과거 데이터 백필 |
| `POST /collect/yfinance-backfill` | yfinance 폴백 백필 |
| `POST /signal/calculate` | 페어 회귀로 한국 종목별 미증시 시그널 산출 |
| `GET /signal/{kr_code}` | 한국 종목 KR 코드별 시그널 |
| `GET /signal/latest` | 전체 종목 최신 시그널 |
| `GET /pairs` | 등록된 KR↔US 페어 |
| `POST /pairs/recalc-beta` | 60일 회귀 베타 재계산 |
| `POST /pairs/discover` | KOSPI200 × S&P500 상관 자동 발굴 |
| `GET /etfs` / `GET /etfs/{etf}/latest` | 섹터 ETF 현황 |
**기여 점수**:
- 섹터 ETF 동조 (SOXX/XBI/LIT 등 14개) → 같은 한국 섹터 ±5점
- 개별 페어 60일 회귀 베타 (NVDA↔SK하이닉스 등) → ±10점
스케줄: 매일 KST 07:30 수집, 08:00 시그널 계산.
---
### `/home/kyu/trading/graph-engine/main.py`
GAT 그래프 신경망 (PyTorch). 포트 9090.
| 엔드포인트 | 역할 |
|-----------|------|
| `POST /graph/build` | 노드/엣지 재구성 |
| `POST /train` | 학습 (6mo rolling) |
| `POST /predict` | 전체 추론 → `stock_scores.graph_score` |
| `GET /predict/{code}` | 종목별 예측 값 |
| `GET /status` | 모델/학습 상태 |
- **노드**: 활성 종목, 12피처 (1d/5d/20d 수익률, vol_ratio, RSI, tech_score, ROE, 영업이익률, 부채비율, 7d 뉴스, 미증시 overnight, log_mcap)
- **엣지**: ① 60일 가격 상관 |corr|≥0.4 ② 동일 섹터 ③ 뉴스 공기 ≥3회
- **스케줄**: 일요 06:00 학습 / 매일 08:30 추론
- **모델 저장**: `/mnt/nas/models/graph/`
---
### `/home/kyu/trading/telegram-bot/main.py`
텔레그램 명령 처리 봇 (python-telegram-bot).
| 명령 | 역할 |
|------|------|
| `/start /help` | 도움말 |
| `/buy /buys` | 강력매수/매수관심 톱 N |
| `/sell /sells` | 매도관심/강력매도 톱 N |
| `/stock <코드>` | 종목 점수·신호 요약 |
| `/deep <코드>` | RAG+EXAONE 심층분석 결과 |
| `/market` | 시장 요약(섹터·매크로) |
권한: `TELEGRAM_CHAT_ID` 일치 채팅만 응답.
---
## 데이터베이스 스키마 (PostgreSQL)
### `news_analysis`
```
id, title, url, source, published_at, hash(16자 UNIQUE)
sentiment(호재/악재/중립), intensity(1~5), primary_stock
affected_stocks(JSONB), reason, investment_action(매수관심/매도관심/관망)
keywords(JSONB), stock_names(JSONB), stock_codes(JSONB)
similar_count, catalyst, analyzed_at, created_at
```
### `stock_technical`
```
stock_code(UNIQUE), stock_name, price
ma5, ma20, ma60, ma120, rsi, macd, macd_signal, macd_hist
bb_upper, bb_mid, bb_lower, pct_b, stoch_k, stoch_d
vol_ratio, tech_score(-100~100), signal(매수/매도/관망)
obv, obv_trend, vwap20, ichimoku(JSONB) ← 새 지표
signals(JSONB), targets(JSONB), analyzed_at
```
### `dart_corps`
```
stock_code(PK), corp_code, corp_name, modify_date, is_active
※ is_active=true 필터 필수 (상장폐지 제외)
```
### `dart_financials`
```
stock_code, corp_code, corp_name, bsns_year, reprt_code
revenue, operating_profit, net_income
total_assets, total_liabilities, total_equity, operating_cashflow
roe, operating_margin, net_margin, debt_ratio, revenue_growth, fcf_ratio
UNIQUE(stock_code, bsns_year, reprt_code)
※ F-Score는 전년 11011 사업보고서 필요 → years≥2 백필
```
### `stock_scores` (주요 컬럼)
```
stock_code, score_date, total_score, recommendation
news_score, dart_score, technical_score, foreign_score, short_score, price_score
trend_score, intrinsic_value, margin_of_safety, earnings_quality
magic_score, f_score, roc_pct, earnings_yield_pct
altman_z, peg, momentum_pct, beneish_score
gpa_pct, g_score, amihud_illiq, market_beta ← 학술 논문 기반 4개
signals(JSONB), buy_votes, sell_votes
position_size_pct, volatility_60d, market_regime_adj, sector
top_reasons, UNIQUE(stock_code, score_date)
```
### `weight_config`
```
config_date(PK), weights(JSONB), period_days, sample_size
※ 최신 row가 calculate_daily_scores에서 자동 로드 → ensemble_bonus 가중
```
### `pricing_model` (D+E 학습 결과 저장)
```
model_date(PK), linear_coef, linear_intercept, linear_r2,
rf_features(JSONB), rf_r2, sample_size, period_days, created_at
※ /learn-pricing 호출 시 갱신, /predict-price 호출 시 최신 row 사용
```
### `news_raw` (수집/분석 분리용 임시 저장소)
```
id, stock_code, title, url, url_hash(UNIQUE), source,
published_at_text, collected_at, processed(BOOL), processed_at
※ 크롤러는 raw만 빠르게 저장 → 야간 cron(process_raw)이 batch로 LLM 분석
```
### 뷰
- `v_recent_signals` — 최근 24시간 호재/악재 (intensity DESC)
- `v_stock_news_count` — 종목별 7일 뉴스 감성 카운트
---
## 환경변수 키 목록 (.env)
```
POSTGRES_HOST=postgres POSTGRES_PORT=5432 POSTGRES_DB=trading_ai POSTGRES_USER=kyu
REDIS_MAX_MEMORY=2gb REDIS_MAXMEMORY_POLICY=allkeys-lru
OLLAMA_NUM_PARALLEL=4 (EXAONE→GPU0, bge-m3→GPU1 자동 분배)
QDRANT_COLLECTION=news_vectors QDRANT_VECTOR_SIZE=1024
TELEGRAM_CHAT_ID=8690666445
KIS_IS_PAPER=true (모의투자 모드)
ECOS_API_KEY=... (한국은행 매크로 — ecos.bok.or.kr/api 무료)
```
---
## 자주 쓰는 명령어
```bash
# 전체 상태
docker compose ps
# 특정 서비스 로그 (실시간)
docker logs trading-news-collector --tail 50 -f
docker logs trading-score-engine --tail 50 -f
docker logs trading-n8n --tail 30
# 재빌드+재배포
docker compose build <service> && docker compose up -d <service>
# PostgreSQL 쿼리 (DB명: trading_ai, user: kyu)
docker exec trading-postgres psql -U kyu -d trading_ai -c "SELECT ..."
# Redis
source /home/kyu/trading/.env && docker exec trading-redis redis-cli -a $REDIS_PASSWORD
# n8n 마이그레이션 확인
docker exec trading-postgres psql -U kyu -d trading_ai -c "SELECT COUNT(*) FROM n8n.migrations;"
# 추천 종목 조회
docker exec trading-postgres psql -U kyu -d trading_ai -c "
SELECT s.stock_code, d.corp_name, s.total_score, s.recommendation,
s.magic_score, s.f_score, s.roc_pct, s.earnings_yield_pct
FROM stock_scores s JOIN dart_corps d ON d.stock_code=s.stock_code
WHERE d.is_active=true AND s.score_date=CURRENT_DATE
ORDER BY s.total_score DESC LIMIT 10;"
# 다년치 사업보고서 백필 (F-Score / PEG용)
curl -X POST 'http://localhost:8888/collect/financials?count=3000&years=10&annual_only=true'
# 섹터 정보 채우기 (dart_corps.sector)
curl -X POST http://localhost:8888/collect/sectors
# 점수 수동 재계산 (전체 활성종목 대상)
curl -X POST http://localhost:8686/score/calculate
# 백테스트 기반 공식 가중치 학습
curl -X POST 'http://localhost:8686/learn-weights?days=90'
# 현재 가중치 조회
curl -s http://localhost:8686/learn-weights
# 종목별 6공식 신호 확인
docker exec trading-postgres psql -U kyu -d trading_ai -c "
SELECT stock_code, stock_name, total_score, signals
FROM stock_scores WHERE score_date=CURRENT_DATE
ORDER BY total_score DESC LIMIT 10;"
```
---
## 주의사항
- **상장폐지 필터**: 모든 추천 쿼리에 `JOIN dart_corps d ON d.stock_code=... WHERE d.is_active=true` 필수
- **fstab**: NAS 마운트는 반드시 한 줄로 작성, `nfsvers=3` 필수
- **GPU**: Ollama 단일 컨테이너가 GPU0(EXAONE 추론)·GPU1(bge-m3 임베딩) 동시 사용. `runtime: nvidia` 필요
- **KIS**: 현재 모의투자 모드 (`KIS_IS_PAPER=true`)
- **바른API**: KRX→Naver 폴백으로 ~3768 종목 로드. 형태소 실패 시 공백분리 폴백
---
## 현재 이슈 / TODO
- [ ] fstab NAS 항목이 두 줄로 분리됨 — sudo로 직접 수정 필요
- [ ] n8n 워크플로우 재구성 필요 (재부팅 후 소실)
- [x] vLLM 제거 → Ollama 단일 운영으로 전환 (EXAONE 3.5 7.8B + bge-m3)
- [x] n8n DB 마이그레이션 오류 수정 완료 (51개 마이그레이션 등록)
- [x] SSL 인증서 자동 갱신 설정 완료 (certbot.timer + deploy hook → nginx reload)
도메인: al/cla/n8/pns/tr.kyleyang.co.kr, 훅: /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
+8
View File
@@ -0,0 +1,8 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8787
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8787", "--workers", "1", "--log-level", "info"]
+8
View File
@@ -0,0 +1,8 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8282
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8282", "--workers", "1", "--log-level", "info"]
+367
View File
@@ -0,0 +1,367 @@
"""
Aux Signal Service (port 8282, 172.30.0.25)
외부 보조 데이터 수집:
- 네이버 종목 integration API → 컨센서스 + 기관/외국인 일별 수급
- 한국은행 ECOS API → USD/KRW 환율, 국고채 10년 금리
"""
import os
import asyncio
import json
from datetime import date, datetime, timedelta
from typing import Optional, List
import asyncpg
import structlog
import httpx
from fastapi import FastAPI, Query, BackgroundTasks
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.cron import CronTrigger
from pytz import timezone
# ─────────────────────────────────────────────────────────────
# 설정
# ─────────────────────────────────────────────────────────────
PG = {
"host": os.getenv("POSTGRES_HOST", "postgres"),
"port": int(os.getenv("POSTGRES_PORT", 5432)),
"database": os.getenv("POSTGRES_DB", "trading_ai"),
"user": os.getenv("POSTGRES_USER", "kyu"),
"password": os.getenv("POSTGRES_PASSWORD", ""),
}
KST = timezone("Asia/Seoul")
ECOS_KEY = os.getenv("ECOS_API_KEY", "")
logger = structlog.get_logger()
app = FastAPI(title="Aux Signal")
pg_pool: Optional[asyncpg.Pool] = None
scheduler = AsyncIOScheduler(timezone=KST)
# ─────────────────────────────────────────────────────────────
# DDL
# ─────────────────────────────────────────────────────────────
DDL = """
CREATE TABLE IF NOT EXISTS analyst_consensus (
stock_code VARCHAR(10) PRIMARY KEY,
target_price BIGINT DEFAULT 0,
recomm_mean DOUBLE PRECISION DEFAULT 0,
create_date DATE,
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS inst_daily_flow (
stock_code VARCHAR(10),
trade_date DATE,
foreign_net BIGINT DEFAULT 0,
inst_net BIGINT DEFAULT 0,
individual_net BIGINT DEFAULT 0,
foreign_hold_ratio DOUBLE PRECISION DEFAULT 0,
close_price BIGINT DEFAULT 0,
change_amount BIGINT DEFAULT 0,
PRIMARY KEY (stock_code, trade_date)
);
CREATE INDEX IF NOT EXISTS idx_flow_stock ON inst_daily_flow(stock_code, trade_date DESC);
CREATE TABLE IF NOT EXISTS macro_daily (
indicator VARCHAR(20),
trade_date DATE,
value DOUBLE PRECISION,
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (indicator, trade_date)
);
"""
# ─────────────────────────────────────────────────────────────
# Helpers
# ─────────────────────────────────────────────────────────────
def _parse_int(v) -> int:
if not v or v in ("-", ""):
return 0
s = str(v).replace(",", "").replace("+", "").replace(" ", "").strip()
try: return int(s)
except Exception: return 0
def _parse_float(v) -> float:
if not v or v in ("-", ""):
return 0.0
s = str(v).replace(",", "").replace("%", "").replace("+", "").replace(" ", "").strip()
try: return float(s)
except Exception: return 0.0
# ─────────────────────────────────────────────────────────────
# 네이버 종목 integration API
# 응답: consensusInfo + dealTrendInfos (60일치 일별 매매동향)
# ─────────────────────────────────────────────────────────────
async def fetch_naver(client: httpx.AsyncClient, code: str) -> Optional[dict]:
url = f"https://m.stock.naver.com/api/stock/{code}/integration"
try:
r = await client.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=15)
if r.status_code != 200:
return None
return r.json()
except Exception as e:
logger.debug("naver.req_err", code=code, err=str(e))
return None
async def save_consensus(conn, code: str, ci: dict):
if not ci:
return False
tp = _parse_int(ci.get("priceTargetMean"))
rm = _parse_float(ci.get("recommMean"))
cd = ci.get("createDate")
cd_date = None
if cd:
try:
cd_date = datetime.strptime(cd, "%Y-%m-%d").date()
except Exception:
cd_date = None
if tp == 0 and rm == 0:
return False
await conn.execute("""
INSERT INTO analyst_consensus (stock_code, target_price, recomm_mean, create_date)
VALUES ($1, $2, $3, $4)
ON CONFLICT (stock_code) DO UPDATE
SET target_price=$2, recomm_mean=$3, create_date=$4, updated_at=NOW()
""", code, tp, rm, cd_date)
return True
async def save_flow(conn, code: str, dt_infos: list) -> int:
saved = 0
for r in dt_infos:
try:
bz = r.get("bizdate")
if not bz: continue
trade_dt = datetime.strptime(bz, "%Y%m%d").date()
await conn.execute("""
INSERT INTO inst_daily_flow
(stock_code, trade_date, foreign_net, inst_net,
individual_net, foreign_hold_ratio,
close_price, change_amount)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (stock_code, trade_date) DO UPDATE
SET foreign_net=$3, inst_net=$4, individual_net=$5,
foreign_hold_ratio=$6, close_price=$7, change_amount=$8
""", code, trade_dt,
_parse_int(r.get("foreignerPureBuyQuant")),
_parse_int(r.get("organPureBuyQuant")),
_parse_int(r.get("individualPureBuyQuant")),
_parse_float(r.get("foreignerHoldRatio")),
_parse_int(r.get("closePrice")),
_parse_int(r.get("compareToPreviousClosePrice")))
saved += 1
except Exception as e:
logger.debug("flow.save_err", code=code, err=str(e))
return saved
async def collect_naver_data(count: int = 500):
"""시총 상위 N개 종목 컨센서스 + 일별 수급 수집"""
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT DISTINCT ON (d.stock_code) d.stock_code,
COALESCE(p.market_cap, 0) AS mc
FROM dart_corps d
LEFT JOIN stock_prices p ON p.stock_code=d.stock_code
WHERE d.is_active=true
ORDER BY d.stock_code, p.collected_at DESC NULLS LAST
""")
top = sorted(rows, key=lambda r: -r["mc"])[:count]
logger.info("naver.collect_start", count=len(top))
cons_saved, flow_saved = 0, 0
async with httpx.AsyncClient() as client:
for i, row in enumerate(top):
code = row["stock_code"]
j = await fetch_naver(client, code)
if j:
async with pg_pool.acquire() as c2:
if await save_consensus(c2, code, j.get("consensusInfo") or {}):
cons_saved += 1
s = await save_flow(c2, code, j.get("dealTrendInfos") or [])
flow_saved += s
# 네이버 rate-limit 회피
if i < len(top) - 1:
await asyncio.sleep(0.3)
logger.info("naver.collect_done", consensus=cons_saved, flow_rows=flow_saved)
return {"consensus": cons_saved, "flow_rows": flow_saved}
# ─────────────────────────────────────────────────────────────
# ECOS 매크로
# 731Y001 → 환율 (0000001=USD/KRW)
# 817Y002 → 시장금리 (010195000=국고채 10년)
# ─────────────────────────────────────────────────────────────
ECOS_BASE = "https://ecos.bok.or.kr/api/StatisticSearch"
ECOS_SERIES = [
# (indicator_name, stat_code, item_code1, freq)
("usdkrw", "731Y001", "0000001", "D"),
("kor_10y", "817Y002", "010195000", "D"),
("kor_3y", "817Y002", "010190000", "D"),
("kospi", "802Y001", "0001000", "D"),
]
async def fetch_ecos_series(client: httpx.AsyncClient, stat: str, item: str,
freq: str = "D", days: int = 90) -> list:
if not ECOS_KEY:
return []
end = date.today().strftime("%Y%m%d")
start = (date.today() - timedelta(days=days)).strftime("%Y%m%d")
url = f"{ECOS_BASE}/{ECOS_KEY}/json/kr/1/200/{stat}/{freq}/{start}/{end}/{item}"
try:
r = await client.get(url, timeout=15)
if r.status_code != 200:
return []
j = r.json()
return (j.get("StatisticSearch") or {}).get("row", []) or []
except Exception as e:
logger.warning("ecos.req_err", stat=stat, item=item, err=str(e))
return []
async def collect_ecos_macro(days: int = 90):
if not ECOS_KEY:
logger.error("ecos.no_key")
return {"saved": 0, "err": "ECOS_API_KEY missing"}
saved = 0
async with httpx.AsyncClient() as client:
for name, stat, item, freq in ECOS_SERIES:
rows = await fetch_ecos_series(client, stat, item, freq, days)
async with pg_pool.acquire() as conn:
for r in rows:
try:
t = r.get("TIME")
v = r.get("DATA_VALUE")
if not t or not v:
continue
dt = datetime.strptime(t, "%Y%m%d").date()
await conn.execute("""
INSERT INTO macro_daily (indicator, trade_date, value)
VALUES ($1, $2, $3)
ON CONFLICT (indicator, trade_date) DO UPDATE
SET value=$3
""", name, dt, float(v))
saved += 1
except Exception as e:
logger.debug("ecos.save_err", err=str(e))
await asyncio.sleep(0.2)
logger.info("ecos.collect_done", saved=saved)
return {"saved": saved}
# ─────────────────────────────────────────────────────────────
# 시작/종료
# ─────────────────────────────────────────────────────────────
@app.on_event("startup")
async def on_start():
global pg_pool
pg_pool = await asyncpg.create_pool(**PG, min_size=2, max_size=10)
async with pg_pool.acquire() as conn:
await conn.execute(DDL)
# 네이버 컨센서스/수급: 매일 평일 16:30 (장 마감 후)
scheduler.add_job(collect_naver_data, CronTrigger(
day_of_week="mon-fri", hour=16, minute=30),
id="naver_collect", replace_existing=True)
# ECOS 매크로: 매일 새벽 5시
scheduler.add_job(collect_ecos_macro, CronTrigger(hour=5),
id="ecos_macro", replace_existing=True)
scheduler.start()
logger.info("aux-signal.started")
@app.on_event("shutdown")
async def on_stop():
if scheduler.running:
scheduler.shutdown()
if pg_pool:
await pg_pool.close()
# ─────────────────────────────────────────────────────────────
# REST API
# ─────────────────────────────────────────────────────────────
@app.get("/health")
async def health():
return {"ok": True, "service": "aux-signal",
"ts": datetime.now(KST).isoformat()}
@app.post("/collect/naver")
async def manual_naver(count: int = Query(default=500, ge=10, le=2000),
bg: BackgroundTasks = None):
if bg:
bg.add_task(collect_naver_data, count)
return {"status": "queued", "count": count}
return await collect_naver_data(count)
@app.post("/collect/macro")
async def manual_macro(days: int = Query(default=90, ge=7, le=365)):
return await collect_ecos_macro(days)
@app.get("/consensus/{code}")
async def get_consensus(code: str):
async with pg_pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT * FROM analyst_consensus WHERE stock_code=$1", code)
return dict(row) if row else {"err": "no data"}
@app.get("/flow/{code}")
async def get_flow(code: str, days: int = Query(default=30, ge=1, le=90)):
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT trade_date, foreign_net, inst_net, individual_net,
foreign_hold_ratio, close_price, change_amount
FROM inst_daily_flow
WHERE stock_code=$1 AND trade_date >= CURRENT_DATE - $2::int
ORDER BY trade_date DESC
""", code, days)
return [dict(r) for r in rows]
@app.get("/macro/latest")
async def macro_latest():
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT DISTINCT ON (indicator) indicator, trade_date, value
FROM macro_daily
ORDER BY indicator, trade_date DESC
""")
return {r["indicator"]: {"date": str(r["trade_date"]),
"value": float(r["value"])}
for r in rows}
@app.get("/macro/{indicator}")
async def macro_series(indicator: str, days: int = Query(default=30, ge=1, le=365)):
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT trade_date, value FROM macro_daily
WHERE indicator=$1 AND trade_date >= CURRENT_DATE - $2::int
ORDER BY trade_date DESC
""", indicator, days)
return [{"date": str(r["trade_date"]), "value": float(r["value"])}
for r in rows]
@app.get("/stats")
async def stats():
async with pg_pool.acquire() as conn:
c = await conn.fetchval("SELECT COUNT(*) FROM analyst_consensus")
f = await conn.fetchrow(
"SELECT COUNT(*) AS rows, COUNT(DISTINCT stock_code) AS codes,"
" MAX(trade_date) AS latest FROM inst_daily_flow")
m = await conn.fetchrow(
"SELECT COUNT(*) AS rows, COUNT(DISTINCT indicator) AS indicators,"
" MAX(trade_date) AS latest FROM macro_daily")
return {"consensus_stocks": c,
"flow": dict(f) if f else {},
"macro": dict(m) if m else {}}
+8
View File
@@ -0,0 +1,8 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
asyncpg==0.29.0
apscheduler==3.10.4
structlog==24.2.0
httpx==0.27.0
orjson==3.10.3
pytz==2024.1
+204
View File
@@ -0,0 +1,204 @@
"""
바른 API FastAPI 서버 v2
- 서버 시작 시 KRX 전체 종목 동적 로딩
- 24시간마다 자동 갱신
- Bareun gRPC 연결
"""
import asyncio, hashlib, os, time
from contextlib import asynccontextmanager
from typing import Optional
import orjson, redis.asyncio as aioredis, structlog
from bareunpy import Tagger
from fastapi import FastAPI, HTTPException, Response
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from prometheus_fastapi_instrumentator import Instrumentator
from pydantic import BaseModel, Field
from stock_loader import load_all_stocks, auto_refresh_stocks
structlog.configure(processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(),
])
logger = structlog.get_logger()
BAREUN_API_KEY = os.getenv("BAREUN_API_KEY", "")
BAREUN_SERVER_HOST = os.getenv("BAREUN_SERVER_HOST", "bareun")
BAREUN_SERVER_PORT = int(os.getenv("BAREUN_SERVER_PORT", "5656"))
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = int(os.getenv("REDIS_PORT", "6379"))
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
REDIS_DB = int(os.getenv("REDIS_DB", "1"))
NEWS_DEDUP_TTL = int(os.getenv("NEWS_DEDUP_TTL", "86400"))
STOPWORDS = {
"","","","","","또한","","","","","아래","관련","통해",
"위해","대해","따라","때문","이후","이전","현재","최근","지난","올해","내년",
"이번","오늘","어제","내일","이날","같은","다른","많은","","가장","매우",
"이미","아직","모든","","전체","일부","국내","해외","글로벌","세계","한국",
"미국","중국","일본","유럽","가운데","한편","다만","특히","실제","여전히",
"앞으로","지속","계속","자체","관계자","","","","","경우","상황",
"","","",
}
FINANCE_KEYWORDS = {
"실적","매출","영업이익","순이익","적자","흑자","손실","이익","투자","배당",
"자사주","유상증자","합병","인수","M&A","IPO","상장","상폐","거래정지","금리",
"기준금리","환율","달러","반도체","배터리","전기차","수소","AI","인공지능",
"클라우드","바이오","의약품","임상","승인","특허","수주","계약","부도","파산",
"부채","시총","코스피","코스닥","나스닥",
}
START_TIME = time.time()
class AppState:
tagger: Optional[Tagger] = None
redis: Optional[aioredis.Redis] = None
stock_map: dict[str, str] = {}
stock_count: int = 0
refresh_task: Optional[asyncio.Task] = None
state = AppState()
@asynccontextmanager
async def lifespan(app: FastAPI):
state.stock_map = await load_all_stocks()
state.stock_count = len(state.stock_map)
logger.info("stocks.ready", count=state.stock_count)
state.refresh_task = asyncio.create_task(auto_refresh_stocks(state, 24))
try:
state.tagger = Tagger(BAREUN_API_KEY, BAREUN_SERVER_HOST, BAREUN_SERVER_PORT)
logger.info("tagger.ok")
except Exception as e:
logger.warning("tagger.failed", error=str(e))
try:
state.redis = aioredis.Redis(host=REDIS_HOST, port=REDIS_PORT,
password=REDIS_PASSWORD, db=REDIS_DB, decode_responses=True,
socket_connect_timeout=5, retry_on_timeout=True)
await state.redis.ping()
logger.info("redis.ok")
except Exception as e:
logger.error("redis.failed", error=str(e))
yield
if state.refresh_task: state.refresh_task.cancel()
if state.redis: await state.redis.aclose()
app = FastAPI(title="바른 API v2", version="2.0.0", lifespan=lifespan)
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
Instrumentator().instrument(app).expose(app, endpoint="/metrics")
class AnalyzeRequest(BaseModel):
title: str; content: str = ""; url: str = ""; source: str = ""; published_at: str = ""
class StockMention(BaseModel):
name: str; code: str; count: int
class AnalyzeResponse(BaseModel):
hash: str; is_duplicate: bool; stocks: list[StockMention]; keywords: list[str]
filtered_text: str; token_count: int; processing_time_ms: float
class BatchRequest(BaseModel):
items: list[AnalyzeRequest]
def news_hash(title, url):
return hashlib.sha256(f"{title.strip()}{url.strip()}".encode()).hexdigest()[:16]
async def is_duplicate(h):
if not state.redis: return False
try:
r = await state.redis.set(f"news:dedup:{h}", "1", ex=NEWS_DEDUP_TTL, nx=True)
return r is None
except: return False
def extract_morphemes(text):
if not state.tagger or not text.strip():
return [w for w in text.split() if len(w) >= 2 and w not in STOPWORDS]
try:
return [t for t, p in state.tagger.pos(text)
if p in ("NNG","NNP","SL") and len(t) >= 2 and t not in STOPWORDS]
except:
return [w for w in text.split() if len(w) >= 2 and w not in STOPWORDS]
def extract_stocks(text):
found = {}
for name, code in state.stock_map.items():
c = text.count(name)
if c > 0: found[name] = StockMention(name=name, code=code, count=c)
return sorted(found.values(), key=lambda x: x.count, reverse=True)
def build_filtered(nouns, stocks):
sn = {s.name for s in stocks}
p = [n for n in nouns if n in sn]
f = [n for n in nouns if n in FINANCE_KEYWORDS]
g = [n for n in nouns if n not in STOPWORDS and len(n) >= 2]
return " ".join(list(dict.fromkeys(p + f + g))[:100])
def _analyze(req):
text = f"{req.title} {req.content}".strip()
h = news_hash(req.title, req.url)
nouns = extract_morphemes(text)
stocks = extract_stocks(text)
kw = list(dict.fromkeys(n for n in nouns if len(n) >= 2))[:50]
ft = build_filtered(nouns, stocks)
return h, stocks, kw, ft
@app.get("/health")
async def health():
tok = state.tagger is not None
rok = False
if state.redis:
try: await state.redis.ping(); rok = True
except: pass
return JSONResponse(content={"status": "ok" if tok else "degraded",
"tagger": "ok" if tok else "unavailable", "redis": "ok" if rok else "error",
"stocks_loaded": state.stock_count, "uptime": round(time.time()-START_TIME,1)})
@app.post("/analyze")
async def analyze(req: AnalyzeRequest):
t = time.perf_counter()
h, stocks, kw, ft = _analyze(req)
dup = await is_duplicate(h)
ms = round((time.perf_counter()-t)*1000, 2)
return Response(content=orjson.dumps(AnalyzeResponse(
hash=h, is_duplicate=dup, stocks=stocks, keywords=kw,
filtered_text=ft, token_count=len(ft.split()), processing_time_ms=ms
).model_dump()), media_type="application/json")
@app.post("/analyze/batch")
async def analyze_batch(req: BatchRequest):
if len(req.items) > 50: raise HTTPException(400, "최대 50개")
t = time.perf_counter()
results = []
for item in req.items:
try:
h, stocks, kw, ft = _analyze(item)
dup = await is_duplicate(h)
results.append({"title":item.title,"hash":h,"is_duplicate":dup,
"stocks":[s.model_dump() for s in stocks],"keywords":kw,
"filtered_text":ft,"token_count":len(ft.split())})
except Exception as e:
results.append({"title":item.title,"error":str(e),"is_duplicate":False})
ms = round((time.perf_counter()-t)*1000, 2)
dups = sum(1 for r in results if r.get("is_duplicate"))
return Response(content=orjson.dumps({"total":len(results),"duplicates":dups,
"processed":len(results)-dups,"elapsed_ms":ms,"results":results}),
media_type="application/json")
@app.get("/stocks")
async def stocks_list():
return JSONResponse(content={"count":len(state.stock_map),
"stocks":[{"name":k,"code":v} for k,v in list(state.stock_map.items())[:500]]})
@app.post("/stocks/refresh")
async def refresh():
m = await load_all_stocks()
if len(m) > 50:
state.stock_map = m; state.stock_count = len(m)
return JSONResponse(content={"status":"ok","count":len(m)})
raise HTTPException(500, "종목 로딩 실패")
@app.delete("/dedup/flush")
async def flush():
if not state.redis: raise HTTPException(503)
keys = await state.redis.keys("news:dedup:*")
if keys: await state.redis.delete(*keys)
return {"deleted": len(keys)}
+24
View File
@@ -0,0 +1,24 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y \
curl \
gcc \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN mkdir -p /app/logs
EXPOSE 5757
CMD ["python", "-m", "uvicorn", "main:app", \
"--host", "0.0.0.0", \
"--port", "5757", \
"--workers", "4", \
"--loop", "uvloop", \
"--http", "httptools", \
"--log-level", "info"]
+136
View File
@@ -0,0 +1,136 @@
"""
한국 주식/금융 전문 용어 사전
형태소 분석 보완용 — 바른 API가 분리하지 못하는 복합어 및 금융 전문어를 정의
"""
# ── 재무제표 / 회계 ─────────────────────────────────────────────────────────────
FINANCIAL_STATEMENT = {
"매출액","매출원가","매출총이익","영업이익","영업손실","영업외수익","영업외비용",
"세전이익","당기순이익","당기순손실","포괄손익","이익잉여금","자본잉여금",
"납입자본","기타자본","비지배지분","총자산","총부채","총자본","유동자산",
"비유동자산","유동부채","비유동부채","현금및현금성자산","단기금융상품",
"매출채권","재고자산","유형자산","무형자산","투자자산","영업활동현금흐름",
"투자활동현금흐름","재무활동현금흐름","잉여현금흐름","FCF","EBITDA","EBIT",
"ROE","ROA","ROI","EPS","BPS","DPS","PER","PBR","PSR","EV/EBITDA","PCR",
"부채비율","유동비율","당좌비율","자기자본비율","이자보상배율","영업이익률",
"순이익률","자산회전율","재고자산회전율","매출채권회전율",
}
# ── 주식 시장 구조 ───────────────────────────────────────────────────────────────
MARKET_STRUCTURE = {
"코스피","코스닥","코넥스","K-OTC","유가증권시장","장외시장","선물시장",
"옵션시장","ETF","ETN","ELS","DLS","리츠","SPAC","공모주","유상증자","무상증자",
"감자","합병","분할","인적분할","물적분할","자사주매입","자사주소각","배당",
"중간배당","특별배당","주식배당","현금배당","우선주","보통주","전환사채","CB",
"신주인수권부사채","BW","교환사채","EB","DR","GDR","ADR","상장","상장폐지",
"거래정지","매매정지","관리종목","투자주의","투자경고","투자위험","단기과열",
"공매도","대차거래","프로그램매매","외국인","기관","개인","세력","수급",
"시가총액","거래량","거래대금","52주최고","52주최저","신고가","신저가",
}
# ── 기술적 분석 지표 ─────────────────────────────────────────────────────────────
TECHNICAL_INDICATORS = {
"이동평균","이평선","골든크로스","데드크로스","MACD","RSI","스토캐스틱",
"볼린저밴드","볼밴","일목균형표","지지","저항","추세선","채널","삼각수렴",
"쐐기형","헤드앤숄더","역헤드앤숄더","더블탑","더블바텀","","갭업","갭다운",
"윗꼬리","아래꼬리","망치형","도지","장악형","상승반전","하락반전",
"거래량폭발","거래량급증","음봉","양봉","캔들","봉차트","분봉","일봉","주봉","월봉",
"오버솔드","오버바우트","과매수","과매도","다이버전스","모멘텀",
}
# ── 기업 이벤트 / 공시 ───────────────────────────────────────────────────────────
CORPORATE_EVENTS = {
"실적발표","어닝시즌","어닝서프라이즈","어닝쇼크","가이던스","실적전망",
"수주","수주잔고","계약체결","MOU","LOI","업무협약","전략적제휴","조인트벤처",
"JV","M&A","인수합병","적대적인수","공개매수","TOB","경영권분쟁","주주행동주의",
"주주총회","정기주총","임시주총","이사회","대표이사교체","CEO교체","감사의견",
"한정의견","의견거절","부적정의견","횡령","배임","회계부정","분식회계","리콜",
"소송","피소","과징금","제재","영업정지","특허침해","임상성공","임상실패",
"FDA승인","식약처승인","NDA","IND","임상1상","임상2상","임상3상",
}
# ── 거시경제 / 정책 ──────────────────────────────────────────────────────────────
MACRO_ECONOMICS = {
"기준금리","금리인상","금리인하","금리동결","통화정책","양적완화","QE","QT",
"테이퍼링","연준","FED","FOMC","한국은행","한은","ECB","BOJ","금통위",
"환율","달러원","원달러","엔화","위안화","유로","달러인덱스","DXY",
"인플레이션","디플레이션","스태그플레이션","CPI","PPI","PCE","GDP","GNP",
"경상수지","무역수지","외환보유고","국채","회사채","하이일드","신용등급",
"무디스","S&P","피치","국제유가","WTI","브렌트","천연가스","LNG","구리","","",
"반도체수급","공급망","서플라이체인","리쇼어링","니어쇼어링","관세","무역전쟁",
"경제성장률","실업률","소비자심리지수","PMI","ISM","주택착공","소매판매",
}
# ── 산업 / 섹터 키워드 ───────────────────────────────────────────────────────────
SECTOR_KEYWORDS = {
# 반도체
"반도체","메모리","낸드","D램","HBM","HBM3","HBM4","파운드리","팹리스",
"TSMC","삼성파운드리","파운드리","EUV","GAA","3나노","2나노","첨단패키징",
"CoWoS","인터포저","칩렛","AI반도체","NPU","GPU","AI가속기",
# 2차전지/전기차
"배터리","2차전지","리튬이온","전고체배터리","양극재","음극재","분리막","전해질",
"전기차","BEV","PHEV","HEV","충전인프라","급속충전","에너지밀도","배터리팩",
"배터리셀","모듈","BMS","리튬","코발트","니켈","망간","황산리튬",
# 바이오/헬스
"바이오","제약","의약품","신약","오리지널","제네릭","바이오시밀러","항체","mRNA",
"세포치료","유전자치료","CAR-T","ADC","이중항체","희귀질환","항암제","치매",
"알츠하이머","비만치료제","GLP-1","당뇨","심혈관","호흡기",
# AI/IT/플랫폼
"인공지능","AI","LLM","GPT","클로드","제미나이","거대언어모델","생성AI","AGI",
"클라우드","AWS","애저","GCP","SaaS","PaaS","IaaS","데이터센터","GPU서버",
"반도체설계","엣지AI","온디바이스","자율주행","ADAS","라이다","카메라모듈",
"로봇","협동로봇","물류로봇","휴머노이드",
# 방산/우주
"방산","방위산업","무기수출","미사일","드론","무인기","위성","우주항공","발사체",
# 건설/부동산
"부동산","아파트","분양","착공","준공","건설수주","PF","프로젝트파이낸싱",
"리츠","REITs","임대","역세권","재개발","재건축",
}
# ── 가격 움직임 표현 (감성 분석용) ───────────────────────────────────────────────
PRICE_MOVEMENT = {
"급등","급락","폭등","폭락","상한가","하한가","강세","약세","반등","반락",
"상승","하락","보합","소폭상승","소폭하락","대폭상승","대폭하락",
"신고가경신","52주신고가","역사적신고가","저점매수","고점매도",
"눌림","눌림목","지지","저항돌파","박스권","횡보","추세전환",
"모멘텀강화","모멘텀약화","수급개선","수급악화","외국인순매수","외국인순매도",
"기관순매수","기관순매도","공매도잔고","공매도비율",
}
# ── 투자 전략 / 버핏 스타일 ──────────────────────────────────────────────────────
INVESTMENT_STRATEGY = {
"가치투자","성장투자","배당투자","퀀트투자","모멘텀투자","역발상투자",
"내재가치","안전마진","경제적해자","모트","경쟁우위","진입장벽","가격결정력",
"브랜드가치","플랫폼효과","네트워크효과","전환비용","규모의경제","비용우위",
"장기투자","복리효과","분산투자","집중투자","포트폴리오","자산배분","리밸런싱",
"저PER","저PBR","고ROE","배당수익률","배당성향","주주환원","자본배분",
"현금창출력","잉여현금흐름","재투자수익률","ROIC","WACC",
}
# 동사/형용사 형태 (형태소 분석에서 VV/VA 태그로 뽑히는 것들)
FINANCE_VERBS: set[str] = {
"급등","급락","폭등","폭락","상승","하락","반등","돌파","이탈","회복",
"초과","달성","하회","상회","급성장","개선","악화","감소","증가",
}
# 전체 통합 사전 (lookup용)
ALL_FINANCE_TERMS: set[str] = (
FINANCIAL_STATEMENT
| MARKET_STRUCTURE
| TECHNICAL_INDICATORS
| CORPORATE_EVENTS
| MACRO_ECONOMICS
| SECTOR_KEYWORDS
| PRICE_MOVEMENT
| INVESTMENT_STRATEGY
)
# 감성 판단에 중요한 우선순위 높은 단어들
HIGH_PRIORITY_TERMS: set[str] = {
"어닝서프라이즈","어닝쇼크","급등","급락","폭등","폭락","상한가","하한가",
"수주","임상성공","임상실패","FDA승인","횡령","배임","분식회계","상장폐지",
"관리종목","거래정지","배당증가","자사주소각","M&A","합병","분할",
"골든크로스","데드크로스","신고가","신저가","52주신고가","52주신저가",
"외국인순매수","기관순매수","공매도잔고감소","실적발표","가이던스상향",
"가이던스하향","CEO교체","대표이사교체","감사의견거절",
}
+250
View File
@@ -0,0 +1,250 @@
"""
바른 API FastAPI 서버 v2
- 서버 시작 시 KRX 전체 종목 동적 로딩
- 24시간마다 자동 갱신
- Bareun gRPC 연결
"""
import asyncio, hashlib, os, re, time
from contextlib import asynccontextmanager
from typing import Optional
import orjson, redis.asyncio as aioredis, structlog
from bareunpy import Tagger
from fastapi import FastAPI, HTTPException, Response
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from prometheus_fastapi_instrumentator import Instrumentator
from pydantic import BaseModel, Field
from stock_loader import load_all_stocks, auto_refresh_stocks
from finance_dict import ALL_FINANCE_TERMS, HIGH_PRIORITY_TERMS, FINANCE_VERBS as FD_VERBS
structlog.configure(processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(),
])
logger = structlog.get_logger()
BAREUN_API_KEY = os.getenv("BAREUN_API_KEY", "")
BAREUN_SERVER_HOST = os.getenv("BAREUN_SERVER_HOST", "bareun")
BAREUN_SERVER_PORT = int(os.getenv("BAREUN_SERVER_PORT", "5656"))
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = int(os.getenv("REDIS_PORT", "6379"))
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
REDIS_DB = int(os.getenv("REDIS_DB", "1"))
NEWS_DEDUP_TTL = int(os.getenv("NEWS_DEDUP_TTL", "86400"))
STOPWORDS = {
"","","","","","또한","","","","","아래","관련","통해",
"위해","대해","따라","때문","이후","이전","현재","최근","지난","올해","내년",
"이번","오늘","어제","내일","이날","같은","다른","많은","","가장","매우",
"이미","아직","모든","","전체","일부","국내","해외","글로벌","세계","한국",
"미국","중국","일본","유럽","가운데","한편","다만","특히","실제","여전히",
"앞으로","지속","계속","자체","관계자","","","","","경우","상황",
"","","",
}
FINANCE_KEYWORDS = ALL_FINANCE_TERMS
FINANCE_VERBS = {
"급등","급락","폭등","폭락","상승","하락","반등","하락세","상승세",
"돌파","이탈","회복","저항","지지","돌파구","매집","매도","매수",
"초과달성","하회","상회","달성","부진","급성장","감소","증가","개선","악화",
} | FD_VERBS
START_TIME = time.time()
class AppState:
tagger: Optional[Tagger] = None
redis: Optional[aioredis.Redis] = None
stock_map: dict[str, str] = {}
stock_count: int = 0
refresh_task: Optional[asyncio.Task] = None
state = AppState()
@asynccontextmanager
async def lifespan(app: FastAPI):
state.stock_map = await load_all_stocks()
state.stock_count = len(state.stock_map)
logger.info("stocks.ready", count=state.stock_count)
state.refresh_task = asyncio.create_task(auto_refresh_stocks(state, 24))
try:
state.tagger = Tagger(BAREUN_API_KEY, BAREUN_SERVER_HOST, BAREUN_SERVER_PORT)
logger.info("tagger.ok")
except Exception as e:
logger.warning("tagger.failed", error=str(e))
try:
state.redis = aioredis.Redis(host=REDIS_HOST, port=REDIS_PORT,
password=REDIS_PASSWORD, db=REDIS_DB, decode_responses=True,
socket_connect_timeout=5, retry_on_timeout=True)
await state.redis.ping()
logger.info("redis.ok")
except Exception as e:
logger.error("redis.failed", error=str(e))
yield
if state.refresh_task: state.refresh_task.cancel()
if state.redis: await state.redis.aclose()
app = FastAPI(title="바른 API v2", version="2.0.0", lifespan=lifespan)
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
Instrumentator().instrument(app).expose(app, endpoint="/metrics")
class AnalyzeRequest(BaseModel):
title: str; content: str = ""; url: str = ""; source: str = ""; published_at: str = ""
class StockMention(BaseModel):
name: str; code: str; count: int
class AnalyzeResponse(BaseModel):
hash: str; is_duplicate: bool; stocks: list[StockMention]; keywords: list[str]
filtered_text: str; token_count: int; processing_time_ms: float
class BatchRequest(BaseModel):
items: list[AnalyzeRequest]
def news_hash(title, url):
return hashlib.sha256(f"{title.strip()}{url.strip()}".encode()).hexdigest()[:16]
async def is_duplicate(h):
if not state.redis: return False
try:
r = await state.redis.set(f"news:dedup:{h}", "1", ex=NEWS_DEDUP_TTL, nx=True)
return r is None
except: return False
def extract_morphemes(text):
if not state.tagger or not text.strip():
return [w for w in text.split() if len(w) >= 2 and w not in STOPWORDS]
try:
result = []
for t, p in state.tagger.pos(text):
if len(t) < 2 or t in STOPWORDS:
continue
if p in ("NNG", "NNP", "SL"):
result.append(t)
elif p in ("VV", "VA", "XR") and t in FINANCE_VERBS:
result.append(t)
return result
except:
return [w for w in text.split() if len(w) >= 2 and w not in STOPWORDS]
def extract_stocks(text):
found = {}
for name, code in state.stock_map.items():
if len(name) < 2:
continue # 1글자 종목명은 오탐 과다 → 제외
if name.isascii():
# 영문/숫자 약칭(KT·SK·DB 등)은 단어경계 강제 (SKT·KTX 오탐 차단)
pat = rf"(?<![A-Za-z0-9]){re.escape(name)}(?![A-Za-z0-9])"
c = len(re.findall(pat, text))
else:
c = text.count(name) # 한글명: 교착어 특성상 부분일치 유지
if c > 0:
found[name] = StockMention(name=name, code=code, count=c)
# 더 긴 종목명에 포함된 짧은 종목명 제거 ('한국' ⊂ '한국전력', 'KT' ⊂ 'KT&G')
names = list(found)
for n in names:
if any(n != m and n in m for m in names):
found.pop(n, None)
return sorted(found.values(), key=lambda x: x.count, reverse=True)
def build_filtered(nouns, stocks):
sn = {s.name for s in stocks}
seen = set()
result = []
# 1순위: 고중요도 금융 이벤트 (어닝서프라이즈, 상한가 등)
for n in nouns:
if n in HIGH_PRIORITY_TERMS and n not in seen:
result.append(n); seen.add(n)
# 2순위: 종목명
for n in nouns:
if n in sn and n not in seen:
result.append(n); seen.add(n)
# 3순위: 전문 금융 용어 사전
for n in nouns:
if n in FINANCE_KEYWORDS and n not in seen:
result.append(n); seen.add(n)
# 4순위: 나머지 의미있는 명사
for n in nouns:
if n not in STOPWORDS and len(n) >= 2 and n not in seen:
result.append(n); seen.add(n)
return " ".join(result[:120])
def scan_finance_terms(text: str) -> list[str]:
"""형태소 분석 없이 원문에서 금융 전문 용어 직접 탐색 (복합어 대응)"""
found = []
for term in ALL_FINANCE_TERMS:
if term in text:
found.append(term)
return found
def _analyze(req):
text = f"{req.title} {req.content}".strip()
h = news_hash(req.title, req.url)
nouns = extract_morphemes(text)
# 원문 직접 스캔으로 형태소 분석이 놓친 복합어 추가
direct_terms = scan_finance_terms(text)
nouns = list(dict.fromkeys(nouns + direct_terms))
stocks = extract_stocks(text)
kw = list(dict.fromkeys(n for n in nouns if len(n) >= 2))[:50]
ft = build_filtered(nouns, stocks)
return h, stocks, kw, ft
@app.get("/health")
async def health():
tok = state.tagger is not None
rok = False
if state.redis:
try: await state.redis.ping(); rok = True
except: pass
return JSONResponse(content={"status": "ok" if tok else "degraded",
"tagger": "ok" if tok else "unavailable", "redis": "ok" if rok else "error",
"stocks_loaded": state.stock_count, "uptime": round(time.time()-START_TIME,1)})
@app.post("/analyze")
async def analyze(req: AnalyzeRequest):
t = time.perf_counter()
h, stocks, kw, ft = _analyze(req)
dup = await is_duplicate(h)
ms = round((time.perf_counter()-t)*1000, 2)
return Response(content=orjson.dumps(AnalyzeResponse(
hash=h, is_duplicate=dup, stocks=stocks, keywords=kw,
filtered_text=ft, token_count=len(ft.split()), processing_time_ms=ms
).model_dump()), media_type="application/json")
@app.post("/analyze/batch")
async def analyze_batch(req: BatchRequest):
if len(req.items) > 50: raise HTTPException(400, "최대 50개")
t = time.perf_counter()
results = []
for item in req.items:
try:
h, stocks, kw, ft = _analyze(item)
dup = await is_duplicate(h)
results.append({"title":item.title,"hash":h,"is_duplicate":dup,
"stocks":[s.model_dump() for s in stocks],"keywords":kw,
"filtered_text":ft,"token_count":len(ft.split())})
except Exception as e:
results.append({"title":item.title,"error":str(e),"is_duplicate":False})
ms = round((time.perf_counter()-t)*1000, 2)
dups = sum(1 for r in results if r.get("is_duplicate"))
return Response(content=orjson.dumps({"total":len(results),"duplicates":dups,
"processed":len(results)-dups,"elapsed_ms":ms,"results":results}),
media_type="application/json")
@app.get("/stocks")
async def stocks_list():
return JSONResponse(content={"count":len(state.stock_map),
"stocks":[{"name":k,"code":v} for k,v in list(state.stock_map.items())[:500]]})
@app.post("/stocks/refresh")
async def refresh():
m = await load_all_stocks()
if len(m) > 50:
state.stock_map = m; state.stock_count = len(m)
return JSONResponse(content={"status":"ok","count":len(m)})
raise HTTPException(500, "종목 로딩 실패")
@app.delete("/dedup/flush")
async def flush():
if not state.redis: raise HTTPException(503)
keys = await state.redis.keys("news:dedup:*")
if keys: await state.redis.delete(*keys)
return {"deleted": len(keys)}
+14
View File
@@ -0,0 +1,14 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
uvloop==0.19.0
httptools==0.6.1
httpx==0.27.0
redis==5.0.4
pydantic==2.7.1
pydantic-settings==2.3.1
bareunpy==1.7.3
python-multipart==0.0.9
orjson==3.10.3
structlog==24.2.0
prometheus-fastapi-instrumentator==7.0.0
tenacity==8.3.0
+162
View File
@@ -0,0 +1,162 @@
"""
KRX 전체 종목 동적 로딩
- 서버 시작 시 1회 로딩
- 매일 자정 자동 갱신
- 실패 시 하드코딩 폴백
"""
import asyncio
import json
import io
import time
from datetime import datetime
import httpx
import structlog
logger = structlog.get_logger()
# 폴백 종목 (네트워크 실패 시 사용)
FALLBACK_STOCKS = {
"코스피": "KOSPI", "코스닥": "KOSDAQ", "코스피200": "KOSPI200",
"삼성전자": "005930", "SK하이닉스": "000660", "LG에너지솔루션": "373220",
"현대차": "005380", "기아": "000270", "셀트리온": "068270",
"카카오": "035720", "네이버": "035420", "NAVER": "035420",
"삼성바이오로직스": "207940", "KB금융": "105560", "POSCO홀딩스": "005490",
"신한지주": "055550", "LG화학": "051910", "삼성SDI": "006400",
"현대모비스": "012330", "하나금융지주": "086790", "SK텔레콤": "017670",
"KT": "030200", "LG전자": "066570", "한화에어로스페이스": "012450",
"삼성물산": "028260", "HD현대중공업": "329180", "한국전력": "015760",
"HMM": "011200", "대한항공": "003490", "카카오뱅크": "323410",
"크래프톤": "259960", "하이브": "352820", "에코프로": "086520",
"알테오젠": "196170", "한미반도체": "042700",
}
# 약칭/별칭
ALIASES = {
"삼전": "005930", "하닉": "000660", "현차": "005380",
"카뱅": "323410", "삼바": "207940", "삼성바이오": "207940",
"한에솔": "012450", "한화에어": "012450", "LG엔솔": "373220",
"SK하닉": "000660", "포홀": "005490", "에프엔에프": "383220",
"현대자동차": "005380", "네이버": "035420",
}
async def fetch_krx_stocks() -> dict[str, str]:
"""KRX에서 전체 상장 종목 가져오기"""
stock_map = {
"코스피": "KOSPI", "코스닥": "KOSDAQ",
"코스피200": "KOSPI200", "KRX": "KRX",
}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Referer": "http://data.krx.co.kr/contents/MDC/MDI/mdiLoader/index.cmd",
}
async with httpx.AsyncClient(timeout=30, headers=headers) as client:
for mkt_id, mkt_name in [("STK", "KOSPI"), ("KSQ", "KOSDAQ")]:
try:
payload = {
"bld": "dbms/MDC/STAT/standard/MDCSTAT01901",
"locale": "ko_KR",
"mktId": mkt_id,
"share": "1",
"csvxls_is498": "false",
}
resp = await client.post(
"http://data.krx.co.kr/comm/bldAttend/getJsonData.cmd",
data=payload,
)
if resp.status_code == 200:
data = resp.json()
items = data.get("OutBlock_1", [])
for item in items:
name = item.get("ISU_ABBRV", "").strip()
code = item.get("ISU_SRT_CD", "").strip()
if name and code and len(code) == 6:
stock_map[name] = code
logger.info("krx.loaded", market=mkt_name, count=len(items))
else:
logger.warning("krx.http_error", market=mkt_name, status=resp.status_code)
except Exception as e:
logger.warning("krx.fetch_failed", market=mkt_name, error=str(e))
# KRX 실패 시 네이버 금융 폴백
if len(stock_map) < 100:
logger.info("krx.fallback_naver", reason="KRX returned too few stocks")
stock_map.update(await fetch_naver_stocks())
return stock_map
async def fetch_naver_stocks() -> dict[str, str]:
"""네이버 금융에서 종목 가져오기 (KRX 폴백)"""
stock_map = {}
headers = {"User-Agent": "Mozilla/5.0"}
import re
async with httpx.AsyncClient(timeout=15, headers=headers) as client:
for sosok in [0, 1]: # 0=KOSPI, 1=KOSDAQ
for page in range(1, 45):
try:
url = f"https://finance.naver.com/sise/sise_market_sum.naver?sosok={sosok}&page={page}"
resp = await client.get(url)
text = resp.content.decode("euc-kr", errors="ignore")
rows = re.findall(
r"main\.naver\?code=(\d{6})[^>]*>([^<]+)</a>", text
)
if not rows:
break
for code, name in rows:
name = name.strip()
if name and code:
stock_map[name] = code
except Exception:
break
logger.info("naver.loaded", count=len(stock_map))
return stock_map
async def load_all_stocks() -> dict[str, str]:
"""전체 종목 로딩 (KRX → 네이버 → 폴백)"""
try:
stock_map = await fetch_krx_stocks()
if len(stock_map) > 100:
stock_map.update(ALIASES)
logger.info("stocks.loaded", total=len(stock_map), source="KRX")
return stock_map
except Exception as e:
logger.warning("stocks.krx_failed", error=str(e))
try:
stock_map = await fetch_naver_stocks()
if len(stock_map) > 100:
stock_map.update({
"코스피": "KOSPI", "코스닥": "KOSDAQ",
"코스피200": "KOSPI200", "KRX": "KRX",
})
stock_map.update(ALIASES)
logger.info("stocks.loaded", total=len(stock_map), source="Naver")
return stock_map
except Exception as e:
logger.warning("stocks.naver_failed", error=str(e))
# 최종 폴백
fallback = {**FALLBACK_STOCKS, **ALIASES}
logger.warning("stocks.using_fallback", total=len(fallback))
return fallback
async def auto_refresh_stocks(state_ref, interval_hours: int = 24):
"""백그라운드에서 종목 목록 자동 갱신"""
while True:
await asyncio.sleep(interval_hours * 3600)
try:
new_map = await load_all_stocks()
if len(new_map) > 50:
state_ref.stock_map = new_map
logger.info("stocks.refreshed", total=len(new_map))
except Exception as e:
logger.warning("stocks.refresh_failed", error=str(e))
+9
View File
@@ -0,0 +1,9 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl unzip && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN mkdir -p /app/data /app/logs
EXPOSE 8888
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8888", "--workers", "1", "--log-level", "info"]
@@ -0,0 +1,51 @@
# docker-compose.yml services: 블록 안에 추가
# ──────────────────────────────────────────
# DART 공시 수집기 172.30.0.17
# ──────────────────────────────────────────
dart-collector:
build:
context: ./dart-collector
dockerfile: Dockerfile
container_name: trading-dart-collector
restart: unless-stopped
ports:
- "8888:8888"
environment:
DART_API_KEY: "${DART_API_KEY}"
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
BAREUN_API_URL: "http://bareunaapi:5757"
OLLAMA_URL: "http://ollama:11434"
VLLM_URL: "http://vllm:8000"
QDRANT_URL: "http://qdrant:6333"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.17
depends_on:
redis:
condition: service_healthy
bareunaapi:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 2G
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8888/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "10"
File diff suppressed because it is too large Load Diff
+11
View File
@@ -0,0 +1,11 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
httpx==0.27.0
redis==5.0.4
asyncpg==0.29.0
apscheduler==3.10.4
orjson==3.10.3
structlog==24.2.0
xmltodict==0.13.0
openpyxl==3.1.2
zipfile36==0.1.3
+7
View File
@@ -0,0 +1,7 @@
FROM python:3.11-slim
WORKDIR /app
RUN pip install fastapi uvicorn asyncpg redis orjson httpx pydantic \
bcrypt "python-jose[cryptography]" email-validator apscheduler
COPY . .
EXPOSE 8989
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8989", "--workers", "1"]
+59
View File
@@ -0,0 +1,59 @@
"""인증 유틸 - bcrypt 해시 + JWT 토큰 (이메일/비밀번호)"""
import os
from datetime import datetime, timedelta, timezone
import bcrypt
from jose import jwt, JWTError
from fastapi import Header, HTTPException, status
JWT_SECRET = os.getenv("JWT_SECRET", "")
JWT_ALGORITHM = "HS256"
JWT_EXPIRE_DAYS = int(os.getenv("JWT_EXPIRE_DAYS", "7"))
if not JWT_SECRET:
raise RuntimeError("JWT_SECRET 환경변수가 설정되지 않았습니다")
def hash_password(plain: str) -> str:
# bcrypt는 72바이트 제한 — 안전을 위해 잘라서 해시
pw_bytes = plain.encode("utf-8")[:72]
return bcrypt.hashpw(pw_bytes, bcrypt.gensalt()).decode("utf-8")
def verify_password(plain: str, hashed: str) -> bool:
try:
pw_bytes = plain.encode("utf-8")[:72]
return bcrypt.checkpw(pw_bytes, hashed.encode("utf-8"))
except Exception:
return False
def create_token(user_id: int, email: str) -> str:
now = datetime.now(timezone.utc)
payload = {
"sub": str(user_id),
"email": email,
"iat": int(now.timestamp()),
"exp": int((now + timedelta(days=JWT_EXPIRE_DAYS)).timestamp()),
}
return jwt.encode(payload, JWT_SECRET, algorithm=JWT_ALGORITHM)
def decode_token(token: str) -> dict:
try:
return jwt.decode(token, JWT_SECRET, algorithms=[JWT_ALGORITHM])
except JWTError as e:
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=f"invalid token: {e}")
async def current_user(authorization: str = Header(default="")) -> dict:
"""Authorization: Bearer <token> → {"id": int, "email": str}"""
if not authorization.lower().startswith("bearer "):
raise HTTPException(status_code=401, detail="missing bearer token")
token = authorization.split(" ", 1)[1].strip()
payload = decode_token(token)
try:
return {"id": int(payload["sub"]), "email": payload.get("email", "")}
except (KeyError, ValueError):
raise HTTPException(status_code=401, detail="malformed token payload")
# 타이밍 공격 방지용 더미 해시 (이메일이 없을 때도 verify를 호출해 응답시간 균일화)
_DUMMY_HASH = bcrypt.hashpw(b"dummy_password_for_timing", bcrypt.gensalt()).decode("utf-8")
def dummy_verify():
"""존재하지 않는 사용자에 대해서도 bcrypt 비용을 동일하게 지불"""
bcrypt.checkpw(b"dummy_password_for_timing", _DUMMY_HASH.encode("utf-8"))
+292
View File
@@ -0,0 +1,292 @@
<!doctype html>
<html lang="ko">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Trading AI · 종목 카드</title>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans KR", sans-serif;
background: #0d1117; color: #e6edf3; line-height: 1.5;
padding: 16px; max-width: 1400px; margin: 0 auto;
}
header {
display: flex; justify-content: space-between; align-items: center;
margin-bottom: 20px; padding-bottom: 16px; border-bottom: 1px solid #30363d;
}
h1 { font-size: 20px; font-weight: 600; }
.regime {
padding: 6px 14px; border-radius: 20px; font-size: 13px; font-weight: 500;
}
.regime.강세 { background: #1f3d1f; color: #4ade80; }
.regime.약세 { background: #3d1f1f; color: #f87171; }
.regime.중립 { background: #2d2d3a; color: #a5a5b8; }
.summary-bar {
display: flex; gap: 12px; margin-bottom: 20px; flex-wrap: wrap;
}
.summary-pill {
background: #161b22; border: 1px solid #30363d; padding: 8px 14px;
border-radius: 6px; font-size: 13px;
}
.summary-pill .v { color: #58a6ff; font-weight: 600; margin-left: 6px; }
.tabs { display: flex; gap: 8px; margin-bottom: 16px; }
.tab {
padding: 8px 18px; background: #161b22; border: 1px solid #30363d;
border-radius: 6px; cursor: pointer; font-size: 14px;
}
.tab.active { background: #1f6feb; border-color: #1f6feb; color: white; }
.cards { display: grid; grid-template-columns: repeat(auto-fill, minmax(380px, 1fr)); gap: 14px; }
.card {
background: #161b22; border: 1px solid #30363d; border-radius: 10px;
padding: 16px; transition: transform .15s, border-color .15s;
}
.card:hover { transform: translateY(-2px); border-color: #58a6ff; }
.card-head { display: flex; justify-content: space-between; align-items: flex-start; margin-bottom: 12px; }
.name { font-size: 17px; font-weight: 600; color: #e6edf3; }
.code { font-size: 12px; color: #7d8590; margin-top: 2px; }
.score-badge {
text-align: right;
}
.score {
font-size: 28px; font-weight: 700; line-height: 1;
}
.score.강력매수 { color: #4ade80; }
.score.매수관심 { color: #58a6ff; }
.score.관망 { color: #a5a5b8; }
.score.매도관심, .score.강력매도 { color: #f87171; }
.rec {
font-size: 11px; padding: 2px 8px; border-radius: 4px;
margin-top: 4px; display: inline-block; font-weight: 500;
}
.rec.강력매수 { background: #1f3d1f; color: #4ade80; }
.rec.매수관심 { background: #1f2d3d; color: #58a6ff; }
.rec.매도관심, .rec.강력매도 { background: #3d1f1f; color: #f87171; }
.price-line {
display: flex; align-items: baseline; gap: 8px; margin-bottom: 14px;
}
.price { font-size: 20px; font-weight: 600; }
.change { font-size: 14px; font-weight: 500; }
.change.up { color: #4ade80; }
.change.down { color: #f87171; }
.change.flat { color: #7d8590; }
.sector-tag {
font-size: 11px; padding: 2px 8px; background: #21262d;
border-radius: 4px; color: #a5a5b8;
}
.target-grid {
display: grid; grid-template-columns: repeat(2, 1fr); gap: 8px;
background: #0d1117; padding: 10px; border-radius: 6px; margin-bottom: 12px;
}
.target-cell { font-size: 12px; }
.target-cell .label { color: #7d8590; margin-bottom: 2px; }
.target-cell .v { font-weight: 600; }
.target-cell .pct { color: #7d8590; margin-left: 4px; font-size: 11px; }
.target-cell .v.t { color: #4ade80; }
.target-cell .v.s { color: #f87171; }
.metrics {
display: grid; grid-template-columns: repeat(3, 1fr); gap: 6px;
font-size: 12px; padding: 10px 0; border-top: 1px solid #21262d; border-bottom: 1px solid #21262d;
margin-bottom: 10px;
}
.m-cell .label { color: #7d8590; font-size: 10px; margin-bottom: 2px; }
.m-cell .v { font-weight: 500; }
.m-cell .v.good { color: #4ade80; }
.m-cell .v.bad { color: #f87171; }
.position-line {
display: flex; justify-content: space-between; padding: 8px 10px;
background: #1c2839; border-radius: 6px; margin-bottom: 10px; font-size: 12px;
}
.pos-label { color: #7d8590; }
.pos-val { color: #58a6ff; font-weight: 600; }
.reasons { font-size: 12px; color: #a5a5b8; line-height: 1.6; }
.reason-item {
padding: 4px 0; padding-left: 14px; position: relative;
}
.reason-item:before { content: "→"; position: absolute; left: 0; color: #58a6ff; }
.empty { text-align: center; color: #7d8590; padding: 60px 20px; font-size: 14px; }
.loader { text-align: center; padding: 40px; color: #7d8590; }
</style>
</head>
<body>
<header>
<h1>📊 Trading AI · 종목 카드</h1>
<div class="regime" id="regime">로딩…</div>
</header>
<div class="summary-bar" id="summary"></div>
<div class="tabs">
<div class="tab active" data-kind="recommendations">🟢 매수 추천</div>
<div class="tab" data-kind="avoid">🔴 회피 종목</div>
</div>
<div id="cards-container">
<div class="loader">불러오는 중…</div>
</div>
<script>
const API = location.origin;
function fmt(n) {
if (n === null || n === undefined) return "-";
if (typeof n !== "number") return String(n);
return n.toLocaleString("ko-KR");
}
function pctClass(v) { return v > 0 ? "up" : v < 0 ? "down" : "flat"; }
function pctSign(v) { return (v > 0 ? "+" : "") + v + "%"; }
function metricCls(v, good) { return v >= good ? "good" : v < 0 ? "bad" : ""; }
async function loadSummary() {
try {
const r = await fetch(`${API}/api/summary`);
const d = await r.json();
document.getElementById("regime").textContent =
`시장 ${d.market_regime} (${d.market_regime_adj > 0 ? "+" : ""}${d.market_regime_adj})`;
document.getElementById("regime").className = "regime " + (d.market_regime || "중립");
document.getElementById("summary").innerHTML = `
<div class="summary-pill">강력매수<span class="v">${d.strong_buy || 0}</span></div>
<div class="summary-pill">매수관심<span class="v">${d.interest_buy || 0}</span></div>
<div class="summary-pill">매도관심<span class="v">${d.interest_sell || 0}</span></div>
<div class="summary-pill">강력매도<span class="v">${d.strong_sell || 0}</span></div>
<div class="summary-pill">7일 호재 비율<span class="v">${d.sentiment_ratio || 0}%</span></div>
<div class="summary-pill">분석 종목<span class="v">${d.stocks_analyzed || 0}</span></div>
`;
} catch (e) { console.error(e); }
}
function renderCard(s) {
const recCls = s.recommendation || "관망";
const ch = s.change_pct || 0;
const mosCls = s.margin_of_safety > 25 ? "good" : s.margin_of_safety < -25 ? "bad" : "";
const eqCls = metricCls(s.earnings_quality || 0, 5);
const reasons = (s.reasons || []).map(r =>
`<div class="reason-item">${r}</div>`).join("");
return `
<div class="card">
<div class="card-head">
<div>
<div class="name">${s.stock_name}</div>
<div class="code">${s.stock_code} · <span class="sector-tag">${s.sector || "기타"}</span></div>
</div>
<div class="score-badge">
<div class="score ${recCls}">${s.score}</div>
<div class="rec ${recCls}">${s.recommendation}</div>
</div>
</div>
<div class="price-line">
<div class="price">${fmt(s.price)}원</div>
<div class="change ${pctClass(ch)}">${pctSign(ch)}</div>
${s.market_cap_eok ? `<div class="sector-tag">시총 ${fmt(s.market_cap_eok)}억</div>` : ""}
</div>
${s.t1 ? `
<div class="target-grid">
<div class="target-cell">
<div class="label">진입가</div>
<div class="v">${fmt(s.entry_price)}원</div>
</div>
<div class="target-cell">
<div class="label">손절</div>
<div class="v s">${fmt(s.stop_loss)}원</div>
</div>
<div class="target-cell">
<div class="label">T1 (50% 매도)</div>
<div class="v t">${fmt(s.t1)}<span class="pct">${pctSign(s.t1_pct)}</span></div>
</div>
<div class="target-cell">
<div class="label">Trailing (ATR×2)</div>
<div class="v">${fmt(s.trailing_stop)}원</div>
</div>
<div class="target-cell">
<div class="label">T2 (30%)</div>
<div class="v t">${fmt(s.t2)}<span class="pct">${pctSign(s.t2_pct)}</span></div>
</div>
<div class="target-cell">
<div class="label">T3 (20%)</div>
<div class="v t">${fmt(s.t3)}<span class="pct">${pctSign(s.t3_pct)}</span></div>
</div>
</div>` : `<div class="loader" style="padding:8px;font-size:11px;">목표가 데이터 없음 (다음 기술분석 cron에서 갱신)</div>`}
${s.position_size_pct ? `
<div class="position-line">
<span class="pos-label">추천 매수 비중</span>
<span class="pos-val">${s.position_size_pct}% · 변동성 ${s.volatility_60d || "-"}%</span>
</div>` : ""}
<div class="metrics">
<div class="m-cell">
<div class="label">ROE</div>
<div class="v ${metricCls(s.roe || 0, 10)}">${s.roe ?? "-"}%</div>
</div>
<div class="m-cell">
<div class="label">영업이익률</div>
<div class="v ${metricCls(s.operating_margin || 0, 10)}">${s.operating_margin ?? "-"}%</div>
</div>
<div class="m-cell">
<div class="label">부채비율</div>
<div class="v ${(s.debt_ratio||100) <= 50 ? "good" : (s.debt_ratio||0) > 80 ? "bad" : ""}">${s.debt_ratio ?? "-"}%</div>
</div>
<div class="m-cell">
<div class="label">PER</div>
<div class="v">${s.per ?? "-"}</div>
</div>
<div class="m-cell">
<div class="label">안전마진</div>
<div class="v ${mosCls}">${s.margin_of_safety ?? "-"}%</div>
</div>
<div class="m-cell">
<div class="label">이익품질</div>
<div class="v ${eqCls}">${s.earnings_quality ?? "-"}</div>
</div>
<div class="m-cell">
<div class="label">외국인지분</div>
<div class="v">${s.foreign_ratio_pct ?? "-"}%</div>
</div>
<div class="m-cell">
<div class="label">공매도비중</div>
<div class="v ${(s.short_weight_pct||0) > 5 ? "bad" : ""}">${s.short_weight_pct ?? "-"}%</div>
</div>
<div class="m-cell">
<div class="label">FCF</div>
<div class="v ${metricCls(s.fcf_ratio || 0, 5)}">${s.fcf_ratio ?? "-"}%</div>
</div>
</div>
${reasons ? `<div class="reasons">${reasons}</div>` : ""}
</div>
`;
}
async function loadCards(kind) {
const c = document.getElementById("cards-container");
c.innerHTML = '<div class="loader">불러오는 중…</div>';
try {
const r = await fetch(`${API}/api/${kind}?days=1&limit=30`);
const data = await r.json();
if (!data.length) {
c.innerHTML = '<div class="empty">해당 카테고리에 종목이 없습니다.</div>';
return;
}
c.innerHTML = `<div class="cards">${data.map(renderCard).join("")}</div>`;
} catch (e) {
c.innerHTML = `<div class="empty">에러: ${e.message}</div>`;
}
}
document.querySelectorAll(".tab").forEach(t => {
t.addEventListener("click", () => {
document.querySelectorAll(".tab").forEach(x => x.classList.remove("active"));
t.classList.add("active");
loadCards(t.dataset.kind);
});
});
loadSummary();
loadCards("recommendations");
setInterval(loadSummary, 60000);
</script>
</body>
</html>
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,26 @@
-- 사용자 인증 + 포트폴리오 (5~20명 지인 대상, JWT 기반)
-- idempotent: 여러 번 실행해도 안전
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
role VARCHAR(20) NOT NULL DEFAULT 'user',
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
last_login_at TIMESTAMPTZ
);
CREATE TABLE IF NOT EXISTS user_portfolio (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
stock_code VARCHAR(10) NOT NULL,
stock_name VARCHAR(100) NOT NULL DEFAULT '',
buy_price INTEGER NOT NULL,
qty INTEGER NOT NULL,
memo TEXT DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_user_portfolio_user ON user_portfolio(user_id);
CREATE INDEX IF NOT EXISTS idx_user_portfolio_user_code ON user_portfolio(user_id, stock_code);
@@ -0,0 +1,9 @@
-- 회원가입 관리자 승인 + 보안 (계정 잠금)
-- ADMIN_EMAILS 환경변수에 매칭되는 이메일은 register 시 자동 승인+admin
ALTER TABLE users
ADD COLUMN IF NOT EXISTS is_approved BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN IF NOT EXISTS failed_login_count INTEGER NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS locked_until TIMESTAMPTZ;
CREATE INDEX IF NOT EXISTS idx_users_pending ON users(is_approved) WHERE is_approved = false;
+177
View File
@@ -0,0 +1,177 @@
# ============================================================
# docker-compose.yml services: 블록 안에 추가
# Phase 1~4 전체 서비스
# ============================================================
# ── Phase 1: 네이버 뉴스 수집기 ──────────────────────────
news-collector:
build:
context: ./news-collector
dockerfile: Dockerfile
container_name: trading-news-collector
restart: unless-stopped
ports:
- "8787:8787"
environment:
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
BAREUN_API_URL: "http://bareunaapi:5757"
OLLAMA_URL: "http://ollama:11434"
VLLM_URL: "http://vllm:8000"
QDRANT_URL: "http://qdrant:6333"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.16
depends_on:
redis:
condition: service_healthy
bareunaapi:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 2G
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8787/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "5"
# ── Phase 2: 한국투자증권 주가 API ──────────────────────
kis-api:
build:
context: ./kis-api
dockerfile: Dockerfile
container_name: trading-kis-api
restart: unless-stopped
ports:
- "8585:8585"
environment:
KIS_APP_KEY: "${KIS_APP_KEY:-}"
KIS_APP_SECRET: "${KIS_APP_SECRET:-}"
KIS_IS_PAPER: "${KIS_IS_PAPER:-true}"
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.18
depends_on:
redis:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8585/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 3: 종목 점수 엔진 ─────────────────────────────
score-engine:
build:
context: ./score-engine
dockerfile: Dockerfile
container_name: trading-score-engine
restart: unless-stopped
ports:
- "8686:8686"
environment:
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.19
depends_on:
redis:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8686/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 4: 대시보드 API ───────────────────────────────
dashboard-api:
build:
context: ./dashboard-api
dockerfile: Dockerfile
container_name: trading-dashboard-api
restart: unless-stopped
ports:
- "8989:8989"
environment:
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.22
depends_on:
redis:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD-SHELL", "true"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
+830
View File
@@ -0,0 +1,830 @@
# ============================================================
# Trading AI System
# GPU 0: RTX 3060 12GB → Ollama (EXAONE 3.5 추론)
# GPU 1: RTX 3070 8GB → Ollama (bge-m3 임베딩)
# ============================================================
x-logging: &default-logging
driver: "json-file"
options:
max-size: "${LOG_MAX_SIZE:-100m}"
max-file: "${LOG_MAX_FILE:-10}"
x-restart: &restart
restart: unless-stopped
networks:
trading-net:
driver: bridge
ipam:
config:
- subnet: "${SUBNET:-172.30.0.0/16}"
gateway: "${GATEWAY:-172.30.0.1}"
volumes:
redis-data:
qdrant-data:
ollama-data:
n8n-data:
bareun-data:
pgdata:
services:
# ── Phase 1: 네이버 뉴스 수집기 ──────────────────────────
news-collector:
build:
context: ./news-collector
dockerfile: Dockerfile
container_name: trading-news-collector
restart: unless-stopped
init: true
ports:
- "8787:8787"
environment:
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
BAREUN_API_URL: "http://bareunaapi:5757"
OLLAMA_URL: "http://ollama:11434"
QDRANT_URL: "http://qdrant:6333"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.16
depends_on:
redis:
condition: service_healthy
bareunaapi:
condition: service_healthy
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 2G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8787/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "5"
# ── Phase 2: 한국투자증권 주가 API ──────────────────────
kis-api:
build:
context: ./kis-api
dockerfile: Dockerfile
container_name: trading-kis-api
restart: unless-stopped
init: true
ports:
- "8585:8585"
environment:
KIWOOM_APP_KEY: "${KIWOOM_APP_KEY}"
KIWOOM_SECRET_KEY: "${KIWOOM_SECRET_KEY}"
KIWOOM_BASE_URL: "${KIWOOM_BASE_URL:-https://api.kiwoom.com}"
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.18
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8585/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 2.5: 기술적 분석 엔진 ──────────────────────────
ta-engine:
build:
context: ./ta-engine
dockerfile: Dockerfile
container_name: trading-ta-engine
restart: unless-stopped
init: true
ports:
- "8484:8484"
environment:
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
OLLAMA_URL: "http://ollama:11434"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.23
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8484/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 3: 종목 점수 엔진 ─────────────────────────────
score-engine:
build:
context: ./score-engine
dockerfile: Dockerfile
container_name: trading-score-engine
restart: unless-stopped
init: true
ports:
- "8686:8686"
environment:
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
TELEGRAM_BOT_TOKEN: "${TELEGRAM_BOT_TOKEN:-}"
TELEGRAM_CHAT_ID: "${TELEGRAM_CHAT_ID:-}"
ECOS_API_KEY: "${ECOS_API_KEY:-}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.19
depends_on:
redis:
condition: service_healthy
ta-engine:
condition: service_healthy
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8686/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 3.7: 텔레그램 양방향 봇 ───────────────────────
telegram-bot:
build:
context: ./telegram-bot
dockerfile: Dockerfile
container_name: trading-telegram-bot
restart: unless-stopped
environment:
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
TELEGRAM_BOT_TOKEN: "${TELEGRAM_BOT_TOKEN}"
TELEGRAM_CHAT_ID: "${TELEGRAM_CHAT_ID}"
OLLAMA_URL: "http://ollama:11434"
EXAONE_MODEL: "exaone3.5:7.8b"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.26
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "1.0"
memory: 512M
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 3.6: 보조 시그널 (컨센서스/기관수급/매크로) ──
aux-signal:
build:
context: ./aux-signal
dockerfile: Dockerfile
container_name: trading-aux-signal
restart: unless-stopped
init: true
ports:
- "8282:8282"
environment:
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
ECOS_API_KEY: "${ECOS_API_KEY:-}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.25
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8282/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 3.5: 미국증시 동조 시그널 ─────────────────────
us-market:
build:
context: ./us-market
dockerfile: Dockerfile
container_name: trading-us-market
restart: unless-stopped
init: true
ports:
- "8383:8383"
environment:
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
FINNHUB_API_KEY: "${FINNHUB_API_KEY:-}"
ALPHAVANTAGE_API_KEY: "${ALPHAVANTAGE_API_KEY:-}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.24
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8383/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 5: 그래프 신경망 (GAT) ────────────────────────
graph-engine:
build:
context: ./graph-engine
dockerfile: Dockerfile
container_name: trading-graph-engine
restart: unless-stopped
init: true
ports:
- "9090:9090"
environment:
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
GRAPH_MODEL_DIR: "/mnt/nas/models/graph"
TZ: "Asia/Seoul"
volumes:
- /mnt/nas/models:/mnt/nas/models
networks:
trading-net:
ipv4_address: 172.30.0.27
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "4.0"
memory: 4G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:9090/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 90s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ── Phase 4: 대시보드 API ───────────────────────────────
dashboard-api:
build:
context: ./dashboard-api
dockerfile: Dockerfile
container_name: trading-dashboard-api
restart: unless-stopped
init: true
ports:
- "8989:8989"
environment:
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
JWT_SECRET: "${JWT_SECRET}"
JWT_EXPIRE_DAYS: "${JWT_EXPIRE_DAYS}"
ADMIN_EMAILS: "${ADMIN_EMAILS}"
TRUSTED_ORIGINS: "${TRUSTED_ORIGINS}"
TZ: "Asia/Seoul"
networks:
trading-net:
ipv4_address: 172.30.0.22
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD-SHELL", "true"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
# ──────────────────────────────────────────
# DART 공시 수집기 172.30.0.17
# ──────────────────────────────────────────
dart-collector:
build:
context: ./dart-collector
dockerfile: Dockerfile
container_name: trading-dart-collector
restart: unless-stopped
init: true
ports:
- "8888:8888"
environment:
DART_API_KEY: "${DART_API_KEY}"
REDIS_HOST: redis
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_PORT: "${POSTGRES_PORT}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
BAREUN_API_URL: "http://bareunaapi:5757"
OLLAMA_URL: "http://ollama:11434"
QDRANT_URL: "http://qdrant:6333"
TZ: "Asia/Seoul"
N8N_EDITOR_BASE_URL: "https://n8.kyleyang.co.kr"
WEBHOOK_URL: "https://n8.kyleyang.co.kr"
N8N_HOST: "0.0.0.0"
N8N_LISTEN_ADDRESS: "0.0.0.0"
networks:
trading-net:
ipv4_address: 172.30.0.17
depends_on:
redis:
condition: service_healthy
bareunaapi:
condition: service_healthy
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "2.0"
memory: 2G
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8888/health')\" 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "10"
# ──────────────────────────────────────────
# ──────────────────────────────────────────
# PostgreSQL 172.30.0.9
# ──────────────────────────────────────────
postgres:
image: postgres:16
container_name: trading-postgres
<<: *restart
ports:
- "55432:5432"
environment:
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
POSTGRES_DB: "${POSTGRES_DB}"
PGDATA: /var/lib/postgresql/data
TZ: "Asia/Seoul"
volumes:
- pgdata:/var/lib/postgresql/data
- ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql:ro
networks:
trading-net:
ipv4_address: 172.30.0.9
deploy:
resources:
limits:
cpus: "2.0"
memory: 2G
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
logging: *default-logging
# Redis 172.30.0.10
# ──────────────────────────────────────────
redis:
image: redis:7.2-alpine
container_name: trading-redis
<<: *restart
command: >
redis-server
--requirepass ${REDIS_PASSWORD}
--maxmemory ${REDIS_MAX_MEMORY:-2gb}
--maxmemory-policy ${REDIS_MAXMEMORY_POLICY:-allkeys-lru}
--save 900 1
--save 300 10
--appendonly yes
--appendfsync everysec
--tcp-keepalive 300
ports:
- "6379:6379"
volumes:
- redis-data:/data
networks:
trading-net:
ipv4_address: 172.30.0.10
deploy:
resources:
limits:
cpus: "2.0"
memory: 1G
healthcheck:
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
logging: *default-logging
# ──────────────────────────────────────────
# Qdrant 172.30.0.11
# ──────────────────────────────────────────
qdrant:
image: qdrant/qdrant:v1.9.2
container_name: trading-qdrant
<<: *restart
ports:
- "6333:6333"
- "6334:6334"
volumes:
- qdrant-data:/qdrant/storage
- ./qdrant-config/config.yaml:/qdrant/config/production.yaml:ro
environment:
QDRANT__SERVICE__HTTP_PORT: 6333
QDRANT__SERVICE__GRPC_PORT: 6334
QDRANT__LOG_LEVEL: INFO
networks:
trading-net:
ipv4_address: 172.30.0.11
deploy:
resources:
limits:
cpus: "4.0"
memory: 512M
healthcheck:
test: ["CMD-SHELL", "true"]
interval: 10s
timeout: 5s
retries: 3
start_period: 10s
logging: *default-logging
# ──────────────────────────────────────────
# Bareun 형태소 엔진 172.30.0.15
# ──────────────────────────────────────────
bareun:
image: bareunai/bareun:latest
container_name: trading-bareun
<<: *restart
ports:
- "5656:5656"
- "9902:9902"
volumes:
- bareun-data:/bareun/var
networks:
trading-net:
ipv4_address: 172.30.0.15
deploy:
resources:
limits:
cpus: "4.0"
memory: 1500M
reservations:
memory: 512M
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:9902/health || exit 0"]
interval: 20s
timeout: 10s
retries: 5
start_period: 30s
logging: *default-logging
# ──────────────────────────────────────────
# 바른 API (FastAPI 래퍼) 172.30.0.12
# ──────────────────────────────────────────
bareunaapi:
build:
context: ./bareunaapi
dockerfile: Dockerfile
container_name: trading-bareunaapi
<<: *restart
ports:
- "5757:5757"
environment:
BAREUN_API_KEY: "${BAREUN_API_KEY}"
BAREUN_SERVER_HOST: bareun
BAREUN_SERVER_PORT: 5656
REDIS_HOST: redis
REDIS_PORT: 6379
REDIS_PASSWORD: "${REDIS_PASSWORD}"
REDIS_DB: 1
NEWS_DEDUP_TTL: "${NEWS_DEDUP_TTL:-86400}"
LOG_LEVEL: INFO
networks:
trading-net:
ipv4_address: 172.30.0.12
depends_on:
redis:
condition: service_healthy
bareun:
condition: service_started
deploy:
resources:
limits:
cpus: "4.0"
memory: 512M
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5757/health"]
interval: 15s
timeout: 10s
retries: 5
start_period: 40s
logging: *default-logging
# ──────────────────────────────────────────
# Ollama 추론+임베딩 (GPU 0+1) 172.30.0.13
# GPU 0: RTX 3060 12GB → EXAONE 3.5 7.8B 추론
# GPU 1: RTX 3070 8GB → bge-m3 임베딩
# ──────────────────────────────────────────
ollama:
image: ollama/ollama:latest
container_name: trading-ollama
<<: *restart
ports:
- "11434:11434"
volumes:
- ollama-data:/root/.ollama
environment:
OLLAMA_NUM_PARALLEL: "${OLLAMA_NUM_PARALLEL:-4}"
OLLAMA_MAX_LOADED_MODELS: "2"
OLLAMA_FLASH_ATTENTION: "1"
CUDA_VISIBLE_DEVICES: "0,1"
networks:
trading-net:
ipv4_address: 172.30.0.13
deploy:
resources:
limits:
cpus: "8.0"
memory: 24G
reservations:
devices:
- driver: nvidia
device_ids: ["0", "1"]
capabilities: [gpu]
healthcheck:
test: ["CMD-SHELL", "true"]
interval: 10s
timeout: 5s
retries: 3
start_period: 10s
logging: *default-logging
# ──────────────────────────────────────────
# n8n 워크플로우 172.30.0.20
# ──────────────────────────────────────────
n8n:
image: n8nio/n8n:latest
container_name: trading-n8n
<<: *restart
ports:
- "5678:5678"
environment:
N8N_BASIC_AUTH_ACTIVE: "true"
N8N_BASIC_AUTH_USER: "${N8N_BASIC_AUTH_USER}"
N8N_BASIC_AUTH_PASSWORD: "${N8N_BASIC_AUTH_PASSWORD}"
N8N_ENCRYPTION_KEY: "${N8N_ENCRYPTION_KEY}"
WEBHOOK_URL: "${N8N_WEBHOOK_URL}"
N8N_LOG_LEVEL: info
N8N_METRICS: "true"
# DB - NAS PostgreSQL
DB_TYPE: postgresdb
DB_POSTGRESDB_HOST: "${POSTGRES_HOST}"
DB_POSTGRESDB_PORT: "${POSTGRES_PORT:-5432}"
DB_POSTGRESDB_DATABASE: "${POSTGRES_DB}"
DB_POSTGRESDB_USER: "${POSTGRES_USER}"
DB_POSTGRESDB_PASSWORD: "${POSTGRES_PASSWORD}"
DB_POSTGRESDB_SCHEMA: n8n
DB_POSTGRESDB_POOL_SIZE: "10"
DB_POSTGRESDB_CONNECT_TIMEOUT: "30000"
# Redis Queue
QUEUE_BULL_REDIS_HOST: redis
QUEUE_BULL_REDIS_PORT: 6379
QUEUE_BULL_REDIS_PASSWORD: "${REDIS_PASSWORD}"
QUEUE_BULL_REDIS_DB: 0
EXECUTIONS_MODE: queue
N8N_GRACEFUL_SHUTDOWN_TIMEOUT: 30
TZ: "${TZ:-Asia/Seoul}"
GENERIC_TIMEZONE: "${TZ:-Asia/Seoul}"
# 내부 서비스 주소 (워크플로우에서 사용)
BAREUNAAPI_URL: "http://bareunaapi:5757"
OLLAMA_URL: "http://ollama:11434"
QDRANT_URL: "http://qdrant:6333"
REDIS_PASSWORD: "${REDIS_PASSWORD}"
POSTGRES_HOST: "${POSTGRES_HOST}"
POSTGRES_DB: "${POSTGRES_DB}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
NEWS_SIMILARITY_THRESHOLD: "${NEWS_SIMILARITY_THRESHOLD:-0.92}"
TELEGRAM_BOT_TOKEN: "${TELEGRAM_BOT_TOKEN:-}"
TELEGRAM_CHAT_ID: "${TELEGRAM_CHAT_ID:-}"
N8N_SECURE_COOKIE: "false"
# 실행 이력 자동 정리 (느린 n8n 해결)
EXECUTIONS_DATA_PRUNE: "true"
EXECUTIONS_DATA_MAX_AGE: 168
EXECUTIONS_DATA_PRUNE_MAX_COUNT: 10000
volumes:
- n8n-data:/home/node/.n8n
- /mnt/nas:/mnt/nas
networks:
trading-net:
ipv4_address: 172.30.0.20
depends_on:
redis:
condition: service_healthy
qdrant:
condition: service_healthy
bareunaapi:
condition: service_healthy
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "4.0"
memory: 1G
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:5678/healthz"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
logging: *default-logging
# ──────────────────────────────────────────
# n8n Worker 172.30.0.21
# ──────────────────────────────────────────
n8n-worker:
image: n8nio/n8n:latest
container_name: trading-n8n-worker
<<: *restart
command: worker
environment:
N8N_ENCRYPTION_KEY: "${N8N_ENCRYPTION_KEY}"
DB_TYPE: postgresdb
DB_POSTGRESDB_HOST: "${POSTGRES_HOST}"
DB_POSTGRESDB_PORT: "${POSTGRES_PORT:-5432}"
DB_POSTGRESDB_DATABASE: "${POSTGRES_DB}"
DB_POSTGRESDB_USER: "${POSTGRES_USER}"
DB_POSTGRESDB_PASSWORD: "${POSTGRES_PASSWORD}"
DB_POSTGRESDB_SCHEMA: n8n
DB_POSTGRESDB_POOL_SIZE: "5"
DB_POSTGRESDB_CONNECT_TIMEOUT: "30000"
QUEUE_BULL_REDIS_HOST: redis
QUEUE_BULL_REDIS_PORT: 6379
QUEUE_BULL_REDIS_PASSWORD: "${REDIS_PASSWORD}"
QUEUE_BULL_REDIS_DB: 0
EXECUTIONS_MODE: queue
TZ: "${TZ:-Asia/Seoul}"
GENERIC_TIMEZONE: "${TZ:-Asia/Seoul}"
BAREUNAAPI_URL: "http://bareunaapi:5757"
OLLAMA_URL: "http://ollama:11434"
QDRANT_URL: "http://qdrant:6333"
REDIS_PASSWORD: "${REDIS_PASSWORD}"
volumes:
- n8n-data:/home/node/.n8n
- /mnt/nas:/mnt/nas
networks:
trading-net:
ipv4_address: 172.30.0.21
depends_on:
- n8n
deploy:
resources:
limits:
cpus: "4.0"
memory: 512M
logging: *default-logging
+11
View File
@@ -0,0 +1,11 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
# torch는 CPU 빌드만 받음 (Ollama가 GPU 점유, 추론은 CPU로 충분)
RUN pip install --no-cache-dir --default-timeout=300 --retries=5 \
--index-url https://download.pytorch.org/whl/cpu torch==2.4.1
RUN pip install --no-cache-dir --default-timeout=180 --retries=5 -r requirements.txt
COPY . .
EXPOSE 9090
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "9090", "--workers", "1", "--log-level", "info"]
+635
View File
@@ -0,0 +1,635 @@
"""
Graph Engine (port 9090, 172.30.0.25)
한국 종목 그래프 신경망 (GAT) — 다음날 수익률 예측 → stock_scores.graph_score
노드: 한국 활성종목 (dart_corps.is_active=true)
피처(12): 1d/5d/20d 수익률, vol_ratio, rsi, tech_score,
roe, operating_margin, debt_ratio, news_7d, us_overnight, log_mcap
엣지: ① 가격 60일 상관 |corr|>0.4 ② 같은 sector ③ 뉴스 공기 ≥3회
학습: 매주 일요일 06:00 (6mo rolling window)
추론: 매일 08:30 → stock_scores.graph_score
"""
import os
import asyncio
import json
import math
from datetime import date, datetime, timedelta
from typing import Optional, List, Tuple
import asyncpg
import structlog
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.functional as F
from fastapi import FastAPI, Query, BackgroundTasks
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.cron import CronTrigger
from pytz import timezone
# ─────────────────────────────────────────────────────────────
# 설정
# ─────────────────────────────────────────────────────────────
PG = {
"host": os.getenv("POSTGRES_HOST", "postgres"),
"port": int(os.getenv("POSTGRES_PORT", 5432)),
"database": os.getenv("POSTGRES_DB", "trading_ai"),
"user": os.getenv("POSTGRES_USER", "kyu"),
"password": os.getenv("POSTGRES_PASSWORD", ""),
}
KST = timezone("Asia/Seoul")
MODEL_DIR = os.getenv("GRAPH_MODEL_DIR", "/mnt/nas/models/graph")
os.makedirs(MODEL_DIR, exist_ok=True)
MODEL_PATH = os.path.join(MODEL_DIR, "gat_latest.pt")
FEATURE_DIM = 12
HIDDEN_DIM = 32
ATT_HEADS = 4
DROPOUT = 0.3
CORR_LOOKBACK = 60
CORR_THRESHOLD = 0.65
TOP_K_NEIGHBORS = 20 # 노드당 가격 상관 엣지 상한 (메모리/속도 캡)
NEWS_LOOKBACK = 14
NEWS_COOC_MIN = 3
TRAIN_WINDOW_DAYS = 180
SAMPLE_STRIDE_DAYS = 5 # 6개월 / 5일 = ~36 학습 샘플
logger = structlog.get_logger()
app = FastAPI(title="Graph Engine (GAT)")
pg_pool: Optional[asyncpg.Pool] = None
scheduler = AsyncIOScheduler(timezone=KST)
device = torch.device("cpu")
DDL = """
ALTER TABLE stock_scores
ADD COLUMN IF NOT EXISTS graph_score DOUBLE PRECISION;
CREATE TABLE IF NOT EXISTS graph_model_meta (
model_date DATE PRIMARY KEY,
train_samples INT,
val_samples INT,
val_loss DOUBLE PRECISION,
edge_count INT,
node_count INT,
hidden_dim INT,
notes TEXT,
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS graph_predictions (
stock_code VARCHAR(10),
predict_date DATE,
pred_return DOUBLE PRECISION,
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (stock_code, predict_date)
);
"""
# ─────────────────────────────────────────────────────────────
# GAT 모델 (torch-geometric 없이 순수 torch)
# ─────────────────────────────────────────────────────────────
class GATLayer(nn.Module):
def __init__(self, in_dim, out_dim, heads, dropout):
super().__init__()
assert out_dim % heads == 0
self.heads = heads
self.head_dim = out_dim // heads
self.W = nn.Linear(in_dim, out_dim, bias=False)
a_src = torch.empty(1, heads, self.head_dim)
a_dst = torch.empty(1, heads, self.head_dim)
nn.init.xavier_uniform_(a_src)
nn.init.xavier_uniform_(a_dst)
self.a_src = nn.Parameter(a_src.squeeze(0))
self.a_dst = nn.Parameter(a_dst.squeeze(0))
nn.init.xavier_uniform_(self.W.weight)
self.dropout = dropout
def forward(self, x, edge_index, edge_weight=None):
N = x.size(0)
h = self.W(x).view(N, self.heads, self.head_dim)
src, dst = edge_index[0], edge_index[1]
# 어텐션 logits per (edge, head)
e_src = (h[src] * self.a_src.unsqueeze(0)).sum(-1)
e_dst = (h[dst] * self.a_dst.unsqueeze(0)).sum(-1)
e = F.leaky_relu(e_src + e_dst, 0.2)
if edge_weight is not None:
e = e + edge_weight.unsqueeze(-1).log().clamp(min=-10)
# dst별 softmax: max-subtract for stability
e_max = torch.full((N, self.heads), -1e9, device=x.device)
e_max = e_max.scatter_reduce(0, dst.unsqueeze(-1).expand(-1, self.heads),
e, reduce="amax", include_self=True)
e_exp = torch.exp(e - e_max[dst])
denom = torch.zeros(N, self.heads, device=x.device).index_add_(0, dst, e_exp)
alpha = e_exp / (denom[dst] + 1e-12)
alpha = F.dropout(alpha, p=self.dropout, training=self.training)
# 메시지 집계
m = h[src] * alpha.unsqueeze(-1)
out = torch.zeros(N, self.heads, self.head_dim, device=x.device)
out = out.index_add_(0, dst, m)
return out.reshape(N, -1)
class GraphNet(nn.Module):
def __init__(self, in_dim=FEATURE_DIM, hidden=HIDDEN_DIM,
heads=ATT_HEADS, dropout=DROPOUT):
super().__init__()
self.gat1 = GATLayer(in_dim, hidden, heads, dropout)
self.gat2 = GATLayer(hidden, hidden, heads, dropout)
self.head = nn.Linear(hidden, 1)
self.dropout = dropout
def forward(self, x, edge_index, edge_weight=None):
x = F.elu(self.gat1(x, edge_index, edge_weight))
x = F.dropout(x, p=self.dropout, training=self.training)
x = F.elu(self.gat2(x, edge_index, edge_weight))
return self.head(x).squeeze(-1)
# ─────────────────────────────────────────────────────────────
# 데이터 로더
# ─────────────────────────────────────────────────────────────
async def load_active_codes(conn) -> List[str]:
rows = await conn.fetch(
"SELECT stock_code FROM dart_corps WHERE is_active=true ORDER BY stock_code")
return [r["stock_code"] for r in rows]
async def load_features(conn, codes: List[str], target_date: date) -> np.ndarray:
"""노드 피처 행렬 (N, 12) — 결측은 0으로."""
N = len(codes)
F_ = FEATURE_DIM
out = np.zeros((N, F_), dtype=np.float32)
idx = {c: i for i, c in enumerate(codes)}
# ── 가격 모멘텀 (1d/5d/20d 수익률), vol_ratio ──
rows = await conn.fetch("""
SELECT stock_code, dt, close_price, volume
FROM stock_ohlcv
WHERE dt BETWEEN $1 AND $2
ORDER BY stock_code, dt
""", target_date - timedelta(days=35), target_date)
df = pd.DataFrame(rows, columns=["stock_code", "dt", "close", "volume"])
if not df.empty:
df["close"] = df["close"].astype(float)
df["volume"] = df["volume"].astype(float)
for code, g in df.groupby("stock_code"):
if code not in idx:
continue
g = g.sort_values("dt")
closes = g["close"].values
vols = g["volume"].values
if len(closes) >= 2:
out[idx[code], 0] = (closes[-1] / closes[-2] - 1) * 100
if len(closes) >= 6:
out[idx[code], 1] = (closes[-1] / closes[-6] - 1) * 100
if len(closes) >= 21:
out[idx[code], 2] = (closes[-1] / closes[-21] - 1) * 100
if len(vols) >= 20:
v20 = vols[-20:].mean()
out[idx[code], 3] = (vols[-1] / v20) if v20 > 0 else 1.0
# ── 기술적 (RSI, tech_score) ──
rows = await conn.fetch("""
SELECT DISTINCT ON (stock_code) stock_code, rsi, tech_score
FROM stock_technical
WHERE analyzed_at::date <= $1
ORDER BY stock_code, analyzed_at DESC
""", target_date)
for r in rows:
if r["stock_code"] in idx:
out[idx[r["stock_code"]], 4] = float(r["rsi"] or 50)
out[idx[r["stock_code"]], 5] = float(r["tech_score"] or 0)
# ── 펀더멘털 (ROE, op_margin, debt_ratio) ──
rows = await conn.fetch("""
SELECT DISTINCT ON (stock_code) stock_code, roe, operating_margin, debt_ratio
FROM dart_financials
WHERE reprt_code='11011'
ORDER BY stock_code, bsns_year DESC
""")
for r in rows:
if r["stock_code"] in idx:
out[idx[r["stock_code"]], 6] = float(r["roe"] or 0)
out[idx[r["stock_code"]], 7] = float(r["operating_margin"] or 0)
out[idx[r["stock_code"]], 8] = float(r["debt_ratio"] or 0)
# ── 뉴스 감성 7일 ──
rows = await conn.fetch("""
SELECT primary_stock,
SUM(CASE sentiment WHEN '호재' THEN intensity
WHEN '악재' THEN -intensity ELSE 0 END)::float AS s
FROM news_analysis
WHERE analyzed_at BETWEEN $1 AND $2
AND primary_stock IS NOT NULL
GROUP BY primary_stock
""", target_date - timedelta(days=7), target_date + timedelta(days=1))
for r in rows:
if r["primary_stock"] in idx:
out[idx[r["primary_stock"]], 9] = float(r["s"])
# ── us_overnight_adj (latest) ──
rows = await conn.fetch("""
SELECT DISTINCT ON (stock_code) stock_code, us_overnight_adj
FROM stock_scores
WHERE score_date <= $1 AND us_overnight_adj IS NOT NULL
ORDER BY stock_code, score_date DESC
""", target_date)
for r in rows:
if r["stock_code"] in idx:
out[idx[r["stock_code"]], 10] = float(r["us_overnight_adj"])
# ── log market cap ──
rows = await conn.fetch("""
SELECT DISTINCT ON (stock_code) stock_code, market_cap
FROM stock_prices
WHERE market_cap IS NOT NULL
ORDER BY stock_code, collected_at DESC
""")
for r in rows:
if r["stock_code"] in idx and r["market_cap"]:
out[idx[r["stock_code"]], 11] = math.log10(float(r["market_cap"]) + 1)
# 표준화 (피처별 z-score, 클리핑)
mu = out.mean(axis=0)
sd = out.std(axis=0) + 1e-6
out = np.clip((out - mu) / sd, -5, 5)
return out.astype(np.float32)
async def load_price_corr_edges(conn, codes: List[str], target_date: date,
lookback: int = CORR_LOOKBACK,
threshold: float = CORR_THRESHOLD
) -> Tuple[np.ndarray, np.ndarray]:
"""가격 상관 엣지. 반환: edge_index (2, E), edge_weight (E,)."""
idx = {c: i for i, c in enumerate(codes)}
rows = await conn.fetch("""
SELECT stock_code, dt, close_price FROM stock_ohlcv
WHERE dt BETWEEN $1 AND $2 AND stock_code = ANY($3)
ORDER BY stock_code, dt
""", target_date - timedelta(days=int(lookback * 1.6)),
target_date, codes)
if not rows:
return np.zeros((2, 0), dtype=np.int64), np.zeros(0, dtype=np.float32)
df = pd.DataFrame(rows, columns=["code", "dt", "close"])
pivot = df.pivot(index="dt", columns="code", values="close").astype(float)
pivot = pivot.tail(lookback)
rets = pivot.pct_change().dropna(how="all")
# 결측치 많은 종목 제거 (절반 이상 결측)
rets = rets.dropna(axis=1, thresh=int(len(rets) * 0.5))
if rets.shape[1] < 2:
return np.zeros((2, 0), dtype=np.int64), np.zeros(0, dtype=np.float32)
rets = rets.fillna(0)
M = rets.values # (T, K)
M = (M - M.mean(axis=0)) / (M.std(axis=0) + 1e-9)
K = M.shape[1]
corr = (M.T @ M) / max(M.shape[0] - 1, 1)
np.fill_diagonal(corr, 0)
abs_corr = np.abs(corr)
# 임계값 필터 + 노드당 top-K 이웃만 유지 (메모리 캡)
mask = abs_corr > threshold
src_set: List[int] = []
dst_set: List[int] = []
w_set: List[float] = []
code_arr = list(rets.columns)
for i in range(K):
cands = np.where(mask[i])[0]
if len(cands) == 0:
continue
if len(cands) > TOP_K_NEIGHBORS:
top_idx = np.argpartition(-abs_corr[i, cands], TOP_K_NEIGHBORS)[:TOP_K_NEIGHBORS]
cands = cands[top_idx]
for j in cands:
src_set.append(idx[code_arr[i]])
dst_set.append(idx[code_arr[j]])
w_set.append(float(abs_corr[i, j]))
if not src_set:
return np.zeros((2, 0), dtype=np.int64), np.zeros(0, dtype=np.float32)
return (np.stack([np.array(src_set, dtype=np.int64),
np.array(dst_set, dtype=np.int64)]),
np.array(w_set, dtype=np.float32))
async def load_sector_edges(conn, codes: List[str]) -> np.ndarray:
"""같은 sector 노드 간 양방향 엣지."""
idx = {c: i for i, c in enumerate(codes)}
rows = await conn.fetch("""
SELECT stock_code, sector FROM dart_corps
WHERE is_active=true AND sector IS NOT NULL AND stock_code = ANY($1)
""", codes)
by_sec = {}
for r in rows:
by_sec.setdefault(r["sector"], []).append(r["stock_code"])
src, dst = [], []
for sec, lst in by_sec.items():
# 섹터당 종목 수가 100개 초과면 비대해지므로 무작위 샘플
if len(lst) > 80:
import random
random.seed(hash(sec) & 0xffff)
lst = random.sample(lst, 80)
for i, a in enumerate(lst):
for b in lst[i+1:]:
if a in idx and b in idx:
src += [idx[a], idx[b]]
dst += [idx[b], idx[a]]
if not src:
return np.zeros((2, 0), dtype=np.int64)
return np.stack([np.array(src, dtype=np.int64), np.array(dst, dtype=np.int64)])
async def load_news_edges(conn, codes: List[str], target_date: date,
lookback_days: int = NEWS_LOOKBACK,
min_count: int = NEWS_COOC_MIN) -> np.ndarray:
"""뉴스 공기관계: 같은 뉴스의 affected_stocks 페어 카운트 ≥ min_count."""
idx = {c: i for i, c in enumerate(codes)}
rows = await conn.fetch("""
SELECT affected_stocks FROM news_analysis
WHERE analyzed_at BETWEEN $1 AND $2
AND jsonb_array_length(affected_stocks) > 1
""", target_date - timedelta(days=lookback_days),
target_date + timedelta(days=1))
from collections import Counter
counter: Counter = Counter()
for r in rows:
codes_in_news = r["affected_stocks"]
if isinstance(codes_in_news, str):
codes_in_news = json.loads(codes_in_news)
codes_in_news = [c for c in codes_in_news if c in idx]
for i, a in enumerate(codes_in_news):
for b in codes_in_news[i+1:]:
key = (a, b) if a < b else (b, a)
counter[key] += 1
src, dst = [], []
for (a, b), cnt in counter.items():
if cnt >= min_count:
src += [idx[a], idx[b]]
dst += [idx[b], idx[a]]
if not src:
return np.zeros((2, 0), dtype=np.int64)
return np.stack([np.array(src, dtype=np.int64), np.array(dst, dtype=np.int64)])
async def build_graph(conn, target_date: date):
"""전체 그래프 구성. 반환: (codes, x, edge_index, edge_weight)."""
codes = await load_active_codes(conn)
x = await load_features(conn, codes, target_date)
ei_corr, ew_corr = await load_price_corr_edges(conn, codes, target_date)
ei_sec = await load_sector_edges(conn, codes)
ei_news = await load_news_edges(conn, codes, target_date)
# 결합 (sector/news는 weight=1.0)
ei_all = np.concatenate([ei_corr, ei_sec, ei_news], axis=1)
ew_all = np.concatenate([
ew_corr,
np.full(ei_sec.shape[1], 0.5, dtype=np.float32),
np.full(ei_news.shape[1], 0.7, dtype=np.float32),
])
logger.info("graph.built", nodes=len(codes), edges=int(ei_all.shape[1]),
price=int(ei_corr.shape[1]), sector=int(ei_sec.shape[1]),
news=int(ei_news.shape[1]))
return codes, x, ei_all.astype(np.int64), ew_all.astype(np.float32)
async def load_labels(conn, codes: List[str], target_date: date) -> np.ndarray:
"""target_date 종가 → target_date+1 영업일 종가의 변화율 (%)."""
idx = {c: i for i, c in enumerate(codes)}
labels = np.zeros(len(codes), dtype=np.float32)
mask = np.zeros(len(codes), dtype=bool)
rows = await conn.fetch("""
SELECT stock_code, dt, close_price FROM stock_ohlcv
WHERE dt BETWEEN $1 AND $2 AND stock_code = ANY($3)
ORDER BY stock_code, dt
""", target_date, target_date + timedelta(days=7), codes)
by_code = {}
for r in rows:
by_code.setdefault(r["stock_code"], []).append(
(r["dt"], float(r["close_price"])))
for code, lst in by_code.items():
lst.sort()
if code in idx and len(lst) >= 2:
t0, c0 = lst[0]
t1, c1 = lst[1]
if c0 > 0:
labels[idx[code]] = (c1 / c0 - 1) * 100
mask[idx[code]] = True
return labels, mask
# ─────────────────────────────────────────────────────────────
# 학습
# ─────────────────────────────────────────────────────────────
async def train_model(window_days: int = TRAIN_WINDOW_DAYS,
stride: int = SAMPLE_STRIDE_DAYS,
epochs: int = 30, lr: float = 1e-3):
"""rolling 윈도우 학습. 일별이 아닌 stride 간격으로 샘플링."""
today = date.today()
sample_dates = [today - timedelta(days=window_days - i)
for i in range(0, window_days, stride)]
sample_dates = [d for d in sample_dates if d.weekday() < 5]
if len(sample_dates) < 8:
return {"err": "too few samples", "samples": len(sample_dates)}
val_split = max(1, len(sample_dates) // 5)
train_dates = sample_dates[:-val_split]
val_dates = sample_dates[-val_split:]
logger.info("train.split", train=len(train_dates), val=len(val_dates))
async with pg_pool.acquire() as conn:
# 모든 샘플 그래프 미리 구성
samples = []
for d in sample_dates:
try:
codes, x, ei, ew = await build_graph(conn, d)
y, m = await load_labels(conn, codes, d)
if m.sum() < 10:
continue
samples.append({
"date": d,
"x": torch.tensor(x),
"ei": torch.tensor(ei),
"ew": torch.tensor(ew),
"y": torch.tensor(y),
"m": torch.tensor(m),
})
except Exception as e:
logger.warning("train.sample_err", date=str(d), err=str(e))
if len(samples) < 4:
return {"err": "no valid samples", "got": len(samples)}
val_count = max(1, len(samples) // 5)
train_set = samples[:-val_count]
val_set = samples[-val_count:]
model = GraphNet().to(device)
opt = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=1e-5)
best_val = float("inf")
for epoch in range(epochs):
model.train()
tot = 0.0
for s in train_set:
opt.zero_grad()
pred = model(s["x"].to(device), s["ei"].to(device),
s["ew"].to(device))
mask = s["m"].to(device)
loss = F.huber_loss(pred[mask], s["y"].to(device)[mask], delta=2.0)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
opt.step()
tot += float(loss)
tot /= len(train_set)
model.eval()
v = 0.0
with torch.no_grad():
for s in val_set:
pred = model(s["x"].to(device), s["ei"].to(device),
s["ew"].to(device))
mask = s["m"].to(device)
v += float(F.huber_loss(pred[mask], s["y"].to(device)[mask],
delta=2.0))
v /= len(val_set)
logger.info("train.epoch", epoch=epoch, train_loss=tot, val_loss=v)
if v < best_val:
best_val = v
torch.save({"state": model.state_dict(),
"feature_dim": FEATURE_DIM,
"hidden": HIDDEN_DIM,
"heads": ATT_HEADS}, MODEL_PATH)
async with pg_pool.acquire() as conn:
await conn.execute("""
INSERT INTO graph_model_meta
(model_date, train_samples, val_samples, val_loss,
edge_count, node_count, hidden_dim, notes)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (model_date) DO UPDATE
SET train_samples=$2, val_samples=$3, val_loss=$4,
edge_count=$5, node_count=$6, hidden_dim=$7, notes=$8
""", today, len(train_set), len(val_set), best_val,
int(samples[-1]["ei"].shape[1]), int(samples[-1]["x"].shape[0]),
HIDDEN_DIM, f"epochs={epochs} lr={lr} stride={stride}")
return {"trained": True, "train_samples": len(train_set),
"val_samples": len(val_set), "best_val_loss": best_val}
# ─────────────────────────────────────────────────────────────
# 추론
# ─────────────────────────────────────────────────────────────
async def predict_today() -> dict:
if not os.path.exists(MODEL_PATH):
return {"err": "no trained model"}
today = date.today()
async with pg_pool.acquire() as conn:
codes, x, ei, ew = await build_graph(conn, today)
model = GraphNet().to(device)
ckpt = torch.load(MODEL_PATH, map_location=device, weights_only=True)
model.load_state_dict(ckpt["state"])
model.eval()
with torch.no_grad():
pred = model(torch.tensor(x).to(device),
torch.tensor(ei).to(device),
torch.tensor(ew).to(device))
pred_np = pred.cpu().numpy()
# graph_predictions만 저장 — score-engine이 calculate_daily_scores 시 읽어 ensemble에 적용.
async with pg_pool.acquire() as conn:
async with conn.transaction():
for i, code in enumerate(codes):
p = float(pred_np[i])
await conn.execute("""
INSERT INTO graph_predictions (stock_code, predict_date, pred_return)
VALUES ($1, $2, $3)
ON CONFLICT (stock_code, predict_date) DO UPDATE
SET pred_return=$3, created_at=NOW()
""", code, today, p)
return {"predicted": len(codes), "date": str(today)}
# ─────────────────────────────────────────────────────────────
# FastAPI
# ─────────────────────────────────────────────────────────────
@app.on_event("startup")
async def on_start():
global pg_pool
pg_pool = await asyncpg.create_pool(**PG, min_size=2, max_size=5)
async with pg_pool.acquire() as conn:
await conn.execute(DDL)
# 16:25 KST: score-engine 16:30 calculate_daily_scores 직전 추론
scheduler.add_job(predict_today, CronTrigger(hour=16, minute=25,
day_of_week="mon-fri"),
id="graph_predict", replace_existing=True)
scheduler.add_job(train_model, CronTrigger(day_of_week="sun", hour=6,
minute=0),
id="graph_train", replace_existing=True)
scheduler.start()
logger.info("graph-engine.started")
@app.on_event("shutdown")
async def on_stop():
if scheduler.running:
scheduler.shutdown()
if pg_pool:
await pg_pool.close()
@app.get("/health")
async def health():
return {"status": "ok",
"model_exists": os.path.exists(MODEL_PATH)}
@app.post("/graph/build")
async def manual_build(target: Optional[str] = None):
d = date.fromisoformat(target) if target else date.today()
async with pg_pool.acquire() as conn:
codes, x, ei, ew = await build_graph(conn, d)
return {"nodes": len(codes), "edges": int(ei.shape[1]),
"feature_shape": list(x.shape), "date": str(d)}
@app.post("/train")
async def manual_train(epochs: int = Query(default=30, ge=1, le=200),
window_days: int = Query(default=180, ge=30, le=365),
stride: int = Query(default=5, ge=1, le=30),
bg: BackgroundTasks = None):
if bg:
bg.add_task(train_model, window_days, stride, epochs)
return {"status": "queued", "epochs": epochs}
return await train_model(window_days, stride, epochs)
@app.post("/predict")
async def manual_predict():
return await predict_today()
@app.get("/predict/{code}")
async def predict_stock(code: str, limit: int = Query(default=30, ge=1, le=180)):
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT predict_date, pred_return
FROM graph_predictions
WHERE stock_code=$1
ORDER BY predict_date DESC LIMIT $2
""", code, limit)
return {"code": code, "history": [dict(r) for r in rows]}
@app.get("/status")
async def status():
async with pg_pool.acquire() as conn:
meta = await conn.fetchrow("""
SELECT * FROM graph_model_meta ORDER BY model_date DESC LIMIT 1
""")
latest_pred = await conn.fetchrow("""
SELECT predict_date, COUNT(*) AS n
FROM graph_predictions
GROUP BY predict_date
ORDER BY predict_date DESC LIMIT 1
""")
return {
"model": dict(meta) if meta else None,
"latest_prediction": dict(latest_pred) if latest_pred else None,
"model_exists": os.path.exists(MODEL_PATH),
}
+12
View File
@@ -0,0 +1,12 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
asyncpg==0.29.0
apscheduler==3.10.4
structlog==24.2.0
orjson==3.10.3
httpx==0.27.0
pandas==2.2.2
numpy==1.26.4
scipy==1.13.1
pytz==2024.1
torch==2.4.1
+150
View File
@@ -0,0 +1,150 @@
-- ============================================================
-- Trading AI - PostgreSQL 테이블 초기화
-- NAS (192.168.0.36:32770) 에서 실행
-- ============================================================
-- 뉴스 분석 결과 테이블
CREATE TABLE IF NOT EXISTS news_analysis (
id SERIAL PRIMARY KEY,
title TEXT NOT NULL,
url TEXT DEFAULT '',
source VARCHAR(100) DEFAULT '',
published_at TIMESTAMP WITH TIME ZONE,
hash VARCHAR(16) UNIQUE NOT NULL,
-- AI 분석 결과
sentiment VARCHAR(10) DEFAULT '중립', -- 호재/악재/중립
intensity INTEGER DEFAULT 0, -- 1~5
primary_stock VARCHAR(10) DEFAULT '', -- 주요 종목코드
affected_stocks JSONB DEFAULT '[]'::jsonb, -- 영향 종목 목록
reason TEXT DEFAULT '', -- 판단 근거
investment_action VARCHAR(20) DEFAULT '관망', -- 매수관심/매도관심/관망
-- 형태소 분석 결과
keywords JSONB DEFAULT '[]'::jsonb,
stock_names JSONB DEFAULT '[]'::jsonb,
stock_codes JSONB DEFAULT '[]'::jsonb,
-- 메타데이터
similar_count INTEGER DEFAULT 0,
analyzed_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- 인덱스
CREATE INDEX IF NOT EXISTS idx_news_hash ON news_analysis(hash);
CREATE INDEX IF NOT EXISTS idx_news_sentiment ON news_analysis(sentiment);
CREATE INDEX IF NOT EXISTS idx_news_primary_stock ON news_analysis(primary_stock);
CREATE INDEX IF NOT EXISTS idx_news_published_at ON news_analysis(published_at DESC);
CREATE INDEX IF NOT EXISTS idx_news_analyzed_at ON news_analysis(analyzed_at DESC);
CREATE INDEX IF NOT EXISTS idx_news_intensity ON news_analysis(intensity DESC);
CREATE INDEX IF NOT EXISTS idx_news_investment_action ON news_analysis(investment_action);
-- 종목별 빠른 조회용 GIN 인덱스
CREATE INDEX IF NOT EXISTS idx_news_affected_stocks ON news_analysis USING GIN (affected_stocks);
CREATE INDEX IF NOT EXISTS idx_news_stock_codes ON news_analysis USING GIN (stock_codes);
-- 유용한 뷰: 최근 호재/악재 요약
CREATE OR REPLACE VIEW v_recent_signals AS
SELECT
sentiment,
intensity,
title,
primary_stock,
affected_stocks,
investment_action,
reason,
published_at,
analyzed_at
FROM news_analysis
WHERE sentiment IN ('호재', '악재')
AND analyzed_at >= NOW() - INTERVAL '24 hours'
ORDER BY intensity DESC, analyzed_at DESC;
-- 기술적 분석 결과 테이블
CREATE TABLE IF NOT EXISTS stock_technical (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) UNIQUE NOT NULL,
stock_name VARCHAR(100) DEFAULT '',
price INTEGER DEFAULT 0,
ma5 FLOAT DEFAULT 0,
ma20 FLOAT DEFAULT 0,
ma60 FLOAT DEFAULT 0,
ma120 FLOAT DEFAULT 0,
rsi FLOAT DEFAULT 50,
macd FLOAT DEFAULT 0,
macd_signal FLOAT DEFAULT 0,
macd_hist FLOAT DEFAULT 0,
bb_upper FLOAT DEFAULT 0,
bb_mid FLOAT DEFAULT 0,
bb_lower FLOAT DEFAULT 0,
pct_b FLOAT DEFAULT 0.5,
stoch_k FLOAT DEFAULT 50,
stoch_d FLOAT DEFAULT 50,
vol_ratio FLOAT DEFAULT 1,
tech_score FLOAT DEFAULT 0,
signal VARCHAR(10) DEFAULT '관망',
signals JSONB DEFAULT '[]'::jsonb,
targets JSONB DEFAULT '{}'::jsonb,
analyzed_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_ta_score ON stock_technical(tech_score DESC);
CREATE INDEX IF NOT EXISTS idx_ta_signal ON stock_technical(signal);
CREATE INDEX IF NOT EXISTS idx_ta_code ON stock_technical(stock_code);
-- 유용한 뷰: 종목별 뉴스 카운트
CREATE OR REPLACE VIEW v_stock_news_count AS
SELECT
primary_stock,
sentiment,
COUNT(*) as count,
AVG(intensity) as avg_intensity,
MAX(analyzed_at) as latest_at
FROM news_analysis
WHERE primary_stock != ''
AND analyzed_at >= NOW() - INTERVAL '7 days'
GROUP BY primary_stock, sentiment
ORDER BY count DESC;
-- DART 기업 정보 테이블
CREATE TABLE IF NOT EXISTS dart_corps (
stock_code VARCHAR(10) PRIMARY KEY,
corp_code VARCHAR(20) NOT NULL,
corp_name VARCHAR(200) NOT NULL,
modify_date VARCHAR(20) DEFAULT '',
is_active BOOLEAN DEFAULT false
);
CREATE INDEX IF NOT EXISTS idx_dart_corps_active ON dart_corps(is_active);
CREATE INDEX IF NOT EXISTS idx_dart_corps_corp_code ON dart_corps(corp_code);
-- DART 재무제표 테이블 (버핏 가치지표 포함)
CREATE TABLE IF NOT EXISTS dart_financials (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) NOT NULL,
corp_code VARCHAR(20),
corp_name VARCHAR(200),
bsns_year VARCHAR(4) NOT NULL,
reprt_code VARCHAR(10) NOT NULL,
reprt_name VARCHAR(50),
revenue BIGINT DEFAULT 0,
operating_profit BIGINT DEFAULT 0,
net_income BIGINT DEFAULT 0,
total_assets BIGINT DEFAULT 0,
total_liabilities BIGINT DEFAULT 0,
total_equity BIGINT DEFAULT 0,
operating_cashflow BIGINT DEFAULT 0,
roe FLOAT DEFAULT 0,
operating_margin FLOAT DEFAULT 0,
net_margin FLOAT DEFAULT 0,
debt_ratio FLOAT DEFAULT 0,
revenue_growth FLOAT DEFAULT 0,
fcf_ratio FLOAT DEFAULT 0,
collected_at TIMESTAMP DEFAULT NOW(),
UNIQUE(stock_code, bsns_year, reprt_code)
);
CREATE INDEX IF NOT EXISTS idx_fin_stock ON dart_financials(stock_code);
CREATE INDEX IF NOT EXISTS idx_fin_year ON dart_financials(bsns_year DESC);
CREATE INDEX IF NOT EXISTS idx_fin_roe ON dart_financials(roe DESC);
+235
View File
@@ -0,0 +1,235 @@
"""
주가 수집 + 매매 시그널 서비스 (네이버 금융 기반)
- API 키 불필요
- 시총 상위 200개 종목 현재가
- 매수/매도 목표가 + 손절가 자동 계산
"""
import asyncio,json,os,re
from datetime import datetime,timedelta
from typing import Optional
import asyncpg,httpx,redis.asyncio as aioredis,structlog
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from fastapi import FastAPI,Query
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
structlog.configure(processors=[structlog.processors.TimeStamper(fmt="iso"),structlog.processors.add_log_level,structlog.processors.JSONRenderer()])
logger=structlog.get_logger()
REDIS_HOST=os.getenv("REDIS_HOST","redis")
REDIS_PASSWORD=os.getenv("REDIS_PASSWORD","")
PG_HOST=os.getenv("POSTGRES_HOST","postgres")
PG_PORT=int(os.getenv("POSTGRES_PORT","5432"))
PG_DB=os.getenv("POSTGRES_DB","trading_ai")
PG_USER=os.getenv("POSTGRES_USER","kyu")
PG_PASS=os.getenv("POSTGRES_PASSWORD","7895123")
HEADERS={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
pg_pool:Optional[asyncpg.Pool]=None
redis_cl:Optional[aioredis.Redis]=None
scheduler=AsyncIOScheduler(timezone="Asia/Seoul")
class Stats:
collected=0;last_run="";errors=0
stats=Stats()
async def init_db():
async with pg_pool.acquire() as c:
await c.execute("""CREATE TABLE IF NOT EXISTS stock_prices(
id SERIAL PRIMARY KEY,stock_code VARCHAR(10) NOT NULL,stock_name VARCHAR(100) DEFAULT '',
price INTEGER DEFAULT 0,change_pct FLOAT DEFAULT 0,change_amount INTEGER DEFAULT 0,
volume BIGINT DEFAULT 0,high INTEGER DEFAULT 0,low INTEGER DEFAULT 0,open_price INTEGER DEFAULT 0,
market_cap BIGINT DEFAULT 0,per FLOAT DEFAULT 0,pbr FLOAT DEFAULT 0,
high_52w INTEGER DEFAULT 0,low_52w INTEGER DEFAULT 0,collected_at TIMESTAMP DEFAULT NOW())""")
await c.execute("CREATE INDEX IF NOT EXISTS idx_sp_code ON stock_prices(stock_code)")
await c.execute("CREATE INDEX IF NOT EXISTS idx_sp_time ON stock_prices(collected_at DESC)")
await c.execute("""CREATE TABLE IF NOT EXISTS trade_signals(
id SERIAL PRIMARY KEY,stock_code VARCHAR(10) NOT NULL,stock_name VARCHAR(100) DEFAULT '',
signal_type VARCHAR(10) NOT NULL,current_price INTEGER DEFAULT 0,target_price INTEGER DEFAULT 0,
stop_loss INTEGER DEFAULT 0,expected_return_pct FLOAT DEFAULT 0,risk_reward_ratio FLOAT DEFAULT 0,
confidence FLOAT DEFAULT 0,reason TEXT DEFAULT '',news_score FLOAT DEFAULT 0,
dart_score FLOAT DEFAULT 0,price_momentum FLOAT DEFAULT 0,created_at TIMESTAMP DEFAULT NOW())""")
await c.execute("CREATE INDEX IF NOT EXISTS idx_ts_code ON trade_signals(stock_code)")
await c.execute("CREATE INDEX IF NOT EXISTS idx_ts_time ON trade_signals(created_at DESC)")
logger.info("db.ok")
async def naver_price(client,code):
try:
r=await client.get(f"https://finance.naver.com/item/main.naver?code={code}",headers=HEADERS,timeout=15)
r.encoding="euc-kr";h=r.text
def ex(p,d="0"):
m=re.search(p,h);return m.group(1).replace(",","").strip() if m else d
name=ex(r'class="wrap_company".*?<h2><a[^>]*>([^<]+)</a>',"")
price=int(ex(r'<p class="no_today">.*?<em.*?<span class="blind">([0-9,]+)</span>'))
cpct=ex(r'class="blind">([0-9.\-+]+)%',"0")
vol=ex(r'<td class="first">.*?<span class="blind">([0-9,]+)</span>')
hi=ex(r'최고.*?<em.*?<span class="blind">([0-9,]+)</span>')
lo=ex(r'최저.*?<em.*?<span class="blind">([0-9,]+)</span>')
cap=ex(r'시가총액.*?<em>([0-9,]+)</em>',"0")
per=ex(r'PER.*?<em>([0-9.]+)</em>',"0")
pbr=ex(r'PBR.*?<em>([0-9.]+)</em>',"0")
h52=ex(r'52주.*?최고.*?([0-9,]+)',"0")
l52=ex(r'52주.*?최저.*?([0-9,]+)',"0")
return {"code":code,"name":name,"price":price,"change_pct":float(cpct),
"volume":int(vol),"high":int(hi),"low":int(lo),"market_cap":int(cap),
"per":float(per),"pbr":float(pbr),"high_52w":int(h52),"low_52w":int(l52),
"timestamp":datetime.now().isoformat()}
except Exception as e:
return None
async def get_stock_codes(client,count=200):
if pg_pool:
try:
rows=await pg_pool.fetch("SELECT stock_code FROM dart_corps LIMIT $1",count)
codes=[r["stock_code"] for r in rows if r["stock_code"]];
if len(codes)>50:return codes[:count]
except:pass
codes=[]
for sosok in [0,1]:
for page in range(1,20):
try:
r=await client.get(f"https://finance.naver.com/sise/sise_market_sum.naver?sosok={sosok}&page={page}",headers=HEADERS,timeout=15)
r.encoding="euc-kr"
found=re.findall(r'main\.naver\?code=(\d{6})',r.text)
if not found:break
codes.extend(found);await asyncio.sleep(0.2)
except:break
if len(codes)>=count:break
return list(dict.fromkeys(codes))[:count]
def calc_signal(p,news_sc,dart_sc):
price=p["price"]
if price<=0:return None
h52=p.get("high_52w",0);l52=p.get("low_52w",0);cpct=p.get("change_pct",0)
total=news_sc*0.4+dart_sc*0.3+(cpct*10)*0.3
total=max(-100,min(100,total))
pos=(price-l52)/(h52-l52) if h52>l52>0 else 0.5
if total>=30:
sig="매수"
tp=int(price+(h52-price)*(0.3+min(total,100)/200)) if h52>price else int(price*1.1)
sl=int(price*(0.92+pos*0.03))
conf=min(95,50+total*0.3+(1-pos)*20)
elif total<=-30:
sig="매도"
tp=int(price-(price-l52)*(0.3+min(abs(total),100)/200)) if l52>0 and l52<price else int(price*0.9)
sl=int(price*1.05)
conf=min(95,50+abs(total)*0.3+pos*20)
else:return None
er=((tp-price)/price)*100 if sig=="매수" else ((price-tp)/price)*100
risk=abs(sl-price);reward=abs(tp-price)
rr=reward/risk if risk>0 else 0
return {"signal_type":sig,"current_price":price,"target_price":tp,"stop_loss":sl,
"expected_return_pct":round(er,2),"risk_reward_ratio":round(rr,2),
"confidence":round(conf,1),"price_momentum":round(cpct,2),
"news_score":round(news_sc,1),"dart_score":round(dart_sc,1)}
async def collect_prices():
logger.info("prices.start")
async with httpx.AsyncClient() as client:
codes=await get_stock_codes(client,200)
ok=0
for code in codes:
p=await naver_price(client,code)
if not p or p["price"]<=0:continue
if redis_cl:await redis_cl.set(f"price:{code}",json.dumps(p,ensure_ascii=False),ex=600)
if pg_pool:
try:
await pg_pool.execute("""INSERT INTO stock_prices(stock_code,stock_name,price,change_pct,
volume,high,low,open_price,market_cap,per,pbr,high_52w,low_52w,collected_at)
VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14)""",
code,p["name"],p["price"],p["change_pct"],p["volume"],p["high"],p["low"],
0,p["market_cap"],p["per"],p["pbr"],p["high_52w"],p["low_52w"],datetime.now())
except:pass
ok+=1;await asyncio.sleep(0.3)
await gen_signals()
stats.collected+=ok;stats.last_run=datetime.now().isoformat()
if redis_cl:await redis_cl.set("prices:last_update",datetime.now().isoformat())
logger.info("prices.done",count=ok)
async def gen_signals():
if not pg_pool:return
async with pg_pool.acquire() as c:
scores=await c.fetch("""SELECT stock_code,stock_name,news_score,dart_score,total_score
FROM stock_scores WHERE score_date=(SELECT MAX(score_date) FROM stock_scores)
AND (total_score>=30 OR total_score<=-30)""")
n=0
for row in scores:
code=row["stock_code"]
if not code or len(code)!=6:continue
if redis_cl:
cached=await redis_cl.get(f"price:{code}")
if not cached:continue
pd=json.loads(cached)
else:continue
sig=calc_signal(pd,row["news_score"],row["dart_score"])
if not sig:continue
reasons=await c.fetch("SELECT reason FROM news_analysis WHERE primary_stock=$1 AND intensity>=2 ORDER BY analyzed_at DESC LIMIT 2",code)
reason=" | ".join([r["reason"][:100] for r in reasons])
await c.execute("""INSERT INTO trade_signals(stock_code,stock_name,signal_type,current_price,
target_price,stop_loss,expected_return_pct,risk_reward_ratio,confidence,reason,
news_score,dart_score,price_momentum) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13)""",
code,row["stock_name"],sig["signal_type"],sig["current_price"],sig["target_price"],
sig["stop_loss"],sig["expected_return_pct"],sig["risk_reward_ratio"],sig["confidence"],
reason,sig["news_score"],sig["dart_score"],sig["price_momentum"])
n+=1
logger.info("signals.done",count=n)
app=FastAPI(title="주가+시그널")
app.add_middleware(CORSMiddleware,allow_origins=["*"],allow_methods=["*"],allow_headers=["*"])
@app.on_event("startup")
async def startup():
global pg_pool,redis_cl
pg_pool=await asyncpg.create_pool(host=PG_HOST,port=PG_PORT,database=PG_DB,user=PG_USER,password=PG_PASS,min_size=2,max_size=5)
redis_cl=aioredis.Redis(host=REDIS_HOST,port=6379,password=REDIS_PASSWORD,db=3,decode_responses=True)
await init_db()
scheduler.add_job(collect_prices,"cron",day_of_week="mon-fri",hour="9-16",minute="*/5",id="prices",replace_existing=True)
scheduler.start();logger.info("started")
@app.on_event("shutdown")
async def shutdown():
scheduler.shutdown()
if pg_pool:await pg_pool.close()
if redis_cl:await redis_cl.aclose()
@app.get("/health")
async def health():
lu=await redis_cl.get("prices:last_update") if redis_cl else ""
return JSONResponse(content={"status":"ok","collected":stats.collected,"last_run":stats.last_run,"last_update":lu or ""})
@app.get("/price/{code}")
async def price(code:str):
if redis_cl:
c=await redis_cl.get(f"price:{code}")
if c:return JSONResponse(content=json.loads(c))
async with httpx.AsyncClient() as cl:
p=await naver_price(cl,code)
if p:return JSONResponse(content=p)
return JSONResponse(content={"error":"not found"},status_code=404)
@app.get("/prices")
async def prices(limit:int=Query(default=50)):
if not redis_cl:return JSONResponse(content={"data":{}})
keys=await redis_cl.keys("price:*")
r={};
for k in keys[:limit]:
c=await redis_cl.get(k)
if c:d=json.loads(c);r[d["code"]]=d
return JSONResponse(content={"count":len(r),"data":r})
@app.get("/signals")
async def signals(days:int=Query(default=7)):
async with pg_pool.acquire() as c:
rows=await c.fetch("SELECT * FROM trade_signals WHERE created_at>=NOW()-INTERVAL '%s days' ORDER BY confidence DESC LIMIT 30"%days)
return [dict(r) for r in rows]
@app.get("/signals/{code}")
async def stock_signals(code:str):
async with pg_pool.acquire() as c:
rows=await c.fetch("SELECT * FROM trade_signals WHERE stock_code=$1 ORDER BY created_at DESC LIMIT 10",code)
return [dict(r) for r in rows]
@app.post("/collect")
async def manual():
asyncio.create_task(collect_prices());return {"status":"started"}
+8
View File
@@ -0,0 +1,8 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8585
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8585", "--workers", "1", "--log-level", "info"]
+902
View File
@@ -0,0 +1,902 @@
"""
키움증권 REST API 기반 주가·수급·공매도 수집 서비스
- ka10001: 현재가 + 재무지표 (PER/PBR/ROE/EPS/BPS/외국인비중/시가총액)
- ka10005: 일봉 OHLCV + 외국인·기관 순매수
- ka10008: 외국인 종목별 매매동향 (일자별 보유비중 변화)
- ka10014: 공매도 추이 (잔고수량·거래비중)
- Redis db=3 price:{code} 형식 유지 (score-engine·ta-engine 호환)
"""
import asyncio, json, os, re
from datetime import datetime, timedelta
from typing import Optional
import asyncpg, httpx, redis.asyncio as aioredis, structlog
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from fastapi import FastAPI, Query
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
structlog.configure(processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(),
])
logger = structlog.get_logger()
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
PG_HOST = os.getenv("POSTGRES_HOST", "postgres")
PG_PORT = int(os.getenv("POSTGRES_PORT", "5432"))
PG_DB = os.getenv("POSTGRES_DB", "trading_ai")
PG_USER = os.getenv("POSTGRES_USER", "kyu")
PG_PASS = os.getenv("POSTGRES_PASSWORD", "")
KIWOOM_APP_KEY = os.getenv("KIWOOM_APP_KEY", "")
KIWOOM_SECRET_KEY = os.getenv("KIWOOM_SECRET_KEY", "")
KIWOOM_BASE_URL = os.getenv("KIWOOM_BASE_URL", "https://api.kiwoom.com")
pg_pool: Optional[asyncpg.Pool] = None
redis_cl: Optional[aioredis.Redis] = None
scheduler = AsyncIOScheduler(timezone="Asia/Seoul")
# ── 키움 토큰 관리 ────────────────────────────────────────
class KiwoomToken:
token: str = ""
expires_at: datetime = datetime.min
async def get(self, client: httpx.AsyncClient) -> str:
if self.token and datetime.now() < self.expires_at:
return self.token
resp = await client.post(
f"{KIWOOM_BASE_URL}/oauth2/token",
json={"grant_type": "client_credentials",
"appkey": KIWOOM_APP_KEY,
"secretkey": KIWOOM_SECRET_KEY},
headers={"Content-Type": "application/json;charset=UTF-8"},
timeout=10)
data = resp.json()
if data.get("return_code", -1) != 0:
raise RuntimeError(f"토큰 발급 실패: {data.get('return_msg')}")
self.token = data["token"]
# expires_dt: "20260508151325" 형식
exp_str = data.get("expires_dt", "")
try:
self.expires_at = datetime.strptime(exp_str, "%Y%m%d%H%M%S") - timedelta(minutes=5)
except Exception:
self.expires_at = datetime.now() + timedelta(hours=23)
logger.info("kiwoom.token.refreshed", expires=exp_str)
return self.token
kiwoom_token = KiwoomToken()
# ── 키움 API 호출 헬퍼 ────────────────────────────────────
async def kiwoom_post(client: httpx.AsyncClient, endpoint: str, api_id: str,
body: dict, cont_yn: str = "N", next_key: str = "",
return_headers: bool = False):
token = await kiwoom_token.get(client)
headers = {
"Content-Type": "application/json;charset=UTF-8",
"Authorization": f"Bearer {token}",
"api-id": api_id,
"cont-yn": cont_yn,
}
if next_key:
headers["next-key"] = next_key
r = await client.post(f"{KIWOOM_BASE_URL}{endpoint}",
headers=headers, json=body, timeout=15)
if return_headers: # 연속조회용 cont-yn/next-key 응답헤더 필요
return r.json(), r.headers
return r.json()
def to_int(v: str) -> int:
try:
return int(str(v).replace(",", "").lstrip("+").replace(" ", "") or 0)
except:
return 0
def to_float(v: str) -> float:
try:
return float(str(v).replace(",", "").lstrip("+").replace(" ", "") or 0)
except:
return 0.0
# ── DB 초기화 ─────────────────────────────────────────────
async def init_db():
async with pg_pool.acquire() as c:
await c.execute("""
CREATE TABLE IF NOT EXISTS stock_prices (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) NOT NULL,
stock_name VARCHAR(100) DEFAULT '',
price INTEGER DEFAULT 0,
change_pct FLOAT DEFAULT 0,
change_amount INTEGER DEFAULT 0,
volume BIGINT DEFAULT 0,
high INTEGER DEFAULT 0,
low INTEGER DEFAULT 0,
open_price INTEGER DEFAULT 0,
market_cap BIGINT DEFAULT 0,
per FLOAT DEFAULT 0,
pbr FLOAT DEFAULT 0,
eps FLOAT DEFAULT 0,
bps FLOAT DEFAULT 0,
roe FLOAT DEFAULT 0,
ev FLOAT DEFAULT 0,
high_52w INTEGER DEFAULT 0,
low_52w INTEGER DEFAULT 0,
foreign_ratio FLOAT DEFAULT 0,
credit_ratio FLOAT DEFAULT 0,
float_shares BIGINT DEFAULT 0,
collected_at TIMESTAMP DEFAULT NOW()
)""")
await c.execute("CREATE INDEX IF NOT EXISTS idx_sp_code ON stock_prices(stock_code)")
await c.execute("CREATE INDEX IF NOT EXISTS idx_sp_time ON stock_prices(collected_at DESC)")
await c.execute("""
CREATE TABLE IF NOT EXISTS stock_ohlcv (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) NOT NULL,
dt DATE NOT NULL,
open_price INTEGER DEFAULT 0,
high_price INTEGER DEFAULT 0,
low_price INTEGER DEFAULT 0,
close_price INTEGER DEFAULT 0,
volume BIGINT DEFAULT 0,
trade_amount BIGINT DEFAULT 0,
foreign_ratio FLOAT DEFAULT 0,
foreign_net BIGINT DEFAULT 0,
institution_net BIGINT DEFAULT 0,
individual_net BIGINT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE(stock_code, dt)
)""")
await c.execute("CREATE INDEX IF NOT EXISTS idx_ohlcv_code_dt ON stock_ohlcv(stock_code, dt DESC)")
await c.execute("""
CREATE TABLE IF NOT EXISTS stock_foreign_flow (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) NOT NULL,
dt DATE NOT NULL,
close_price INTEGER DEFAULT 0,
change_qty BIGINT DEFAULT 0,
hold_qty BIGINT DEFAULT 0,
hold_ratio FLOAT DEFAULT 0,
limit_ratio FLOAT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE(stock_code, dt)
)""")
await c.execute("CREATE INDEX IF NOT EXISTS idx_ff_code_dt ON stock_foreign_flow(stock_code, dt DESC)")
await c.execute("""
CREATE TABLE IF NOT EXISTS stock_short_sale (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) NOT NULL,
dt DATE NOT NULL,
close_price INTEGER DEFAULT 0,
short_qty BIGINT DEFAULT 0,
short_balance_qty BIGINT DEFAULT 0,
trade_weight FLOAT DEFAULT 0,
short_avg_price INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE(stock_code, dt)
)""")
await c.execute("CREATE INDEX IF NOT EXISTS idx_ss_code_dt ON stock_short_sale(stock_code, dt DESC)")
await c.execute("""
CREATE TABLE IF NOT EXISTS trade_signals (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) NOT NULL,
stock_name VARCHAR(100) DEFAULT '',
signal_type VARCHAR(10) NOT NULL,
current_price INTEGER DEFAULT 0,
target_price INTEGER DEFAULT 0,
stop_loss INTEGER DEFAULT 0,
expected_return_pct FLOAT DEFAULT 0,
risk_reward_ratio FLOAT DEFAULT 0,
confidence FLOAT DEFAULT 0,
reason TEXT DEFAULT '',
news_score FLOAT DEFAULT 0,
dart_score FLOAT DEFAULT 0,
price_momentum FLOAT DEFAULT 0,
foreign_net_5d BIGINT DEFAULT 0,
short_weight FLOAT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW()
)""")
await c.execute("CREATE INDEX IF NOT EXISTS idx_ts_code ON trade_signals(stock_code)")
await c.execute("CREATE INDEX IF NOT EXISTS idx_ts_time ON trade_signals(created_at DESC)")
logger.info("kiwoom.db.initialized")
# ── 수집 함수 ─────────────────────────────────────────────
async def fetch_basic_info(client: httpx.AsyncClient, code: str, sem: asyncio.Semaphore) -> Optional[dict]:
"""ka10001: 현재가 + 재무지표 (PER/PBR/ROE/EPS/BPS/외국인비중/시가총액)"""
async with sem:
try:
d = await kiwoom_post(client, "/api/dostk/stkinfo", "ka10001", {"stk_cd": code})
if d.get("return_code", -1) != 0:
return None
return {
"code": code,
"name": d.get("stk_nm", ""),
"price": abs(to_int(d.get("cur_prc", "0"))),
"change_amount": to_int(d.get("pred_pre", "0")),
"change_pct": to_float(d.get("flu_rt", "0")),
"volume": to_int(d.get("trde_qty", "0")),
"open_price": abs(to_int(d.get("open_pric", "0"))),
"high": abs(to_int(d.get("high_pric", "0"))),
"low": abs(to_int(d.get("low_pric", "0"))),
"market_cap": to_int(d.get("mac", "0")) * 100_000_000, # 억 → 원
"per": to_float(d.get("per", "0")),
"pbr": to_float(d.get("pbr", "0")),
"eps": to_float(d.get("eps", "0")),
"bps": to_float(d.get("bps", "0")),
"roe": to_float(d.get("roe", "0")),
"ev": to_float(d.get("ev", "0")),
"high_52w": abs(to_int(d.get("250hgst", "0"))),
"low_52w": abs(to_int(d.get("250lwst", "0"))),
"foreign_ratio": to_float(d.get("for_exh_rt", "0")),
"credit_ratio": to_float(d.get("crd_rt", "0")),
"float_shares": to_int(d.get("flo_stk", "0")),
"sale_amt": to_int(d.get("sale_amt", "0")),
"operating_profit": to_int(d.get("bus_pro", "0")),
"net_income": to_int(d.get("cup_nga", "0")),
"timestamp": datetime.now().isoformat(),
}
except Exception as e:
logger.debug("ka10001.err", code=code, error=str(e))
return None
async def fetch_ohlcv(client: httpx.AsyncClient, code: str, sem: asyncio.Semaphore, days: int = 365) -> list:
"""ka10005: 일봉 OHLCV + 외국인·기관 순매수"""
async with sem:
try:
today = datetime.now().strftime("%Y%m%d")
result = []
cont_yn, next_key = "N", ""
for _ in range(20): # 안전 상한(20p ≈ 365봉 충분)
d, hdr = await kiwoom_post(
client, "/api/dostk/mrkcond", "ka10005",
{"stk_cd": code, "dt": today, "req_cnt": days},
cont_yn=cont_yn, next_key=next_key, return_headers=True)
for r in d.get("stk_ddwkmm", []):
cp = abs(to_int(r.get("close_pric", "0")))
if cp <= 0:
continue
result.append({
"dt": r.get("date", ""),
"open": abs(to_int(r.get("open_pric", "0"))),
"high": abs(to_int(r.get("high_pric", "0"))),
"low": abs(to_int(r.get("low_pric", "0"))),
"close": cp,
"volume": to_int(r.get("trde_qty", "0")),
"trade_amount": to_int(r.get("trde_prica", "0")),
"foreign_ratio": to_float(r.get("for_poss", "0")),
"foreign_net": to_int(r.get("for_netprps", "0")),
"institution_net": to_int(r.get("orgn_netprps", "0")),
"individual_net": to_int(r.get("ind_netprps", "0")),
})
# 키움 연속조회: 응답 cont-yn=Y + next-key 있으면 다음 페이지
if (len(result) >= days or hdr.get("cont-yn") != "Y"
or not hdr.get("next-key")):
break
cont_yn, next_key = "Y", hdr.get("next-key", "")
return result
except Exception as e:
logger.debug("ka10005.err", code=code, error=str(e))
return []
async def fetch_foreign_flow(client: httpx.AsyncClient, code: str, sem: asyncio.Semaphore) -> list:
"""ka10008: 외국인 종목별 매매동향"""
async with sem:
try:
d = await kiwoom_post(client, "/api/dostk/frgnistt", "ka10008", {"stk_cd": code})
rows = d.get("stk_frgnr", [])
result = []
for r in rows:
result.append({
"dt": r.get("dt", ""),
"close_price": abs(to_int(r.get("close_pric", "0"))),
"change_qty": to_int(r.get("chg_qty", "0")),
"hold_qty": to_int(r.get("poss_stkcnt", "0")),
"hold_ratio": to_float(r.get("wght", "0")),
"limit_ratio": to_float(r.get("limit_exh_rt", "0")),
})
return result
except Exception as e:
logger.debug("ka10008.err", code=code, error=str(e))
return []
async def fetch_minute_chart(client: httpx.AsyncClient, code: str, tic_scope: str = "1") -> list:
"""ka10080: 주식분봉차트조회. tic_scope: 1, 3, 5, 10, 15, 30, 45, 60 분"""
try:
d = await kiwoom_post(client, "/api/dostk/chart", "ka10080",
{"stk_cd": code, "tic_scope": tic_scope, "upd_stkpc_tp": "1"})
if d.get("return_code", -1) != 0:
return []
rows = d.get("stk_min_pole_chart_qry", []) or d.get("stk_min_pole", []) or []
result = []
for r in rows:
result.append({
"dt": r.get("cntr_tm", r.get("dt", "")),
"open": abs(to_int(r.get("open_pric", "0"))),
"high": abs(to_int(r.get("high_pric", "0"))),
"low": abs(to_int(r.get("low_pric", "0"))),
"close": abs(to_int(r.get("cur_prc", r.get("close_pric", "0")))),
"volume": to_int(r.get("trde_qty", "0")),
})
return result
except Exception as e:
logger.debug("ka10080.err", code=code, error=str(e))
return []
async def fetch_orderbook(client: httpx.AsyncClient, code: str) -> dict:
"""ka10004: 호가잔량 (10단계 매수/매도). path=/api/dostk/mrkcond"""
try:
d = await kiwoom_post(client, "/api/dostk/mrkcond", "ka10004", {"stk_cd": code})
if d.get("return_code", -1) != 0:
return {}
bid, ask = [], []
# 매도 호가 1~10 (sel_1th_pre_bid = 가격, sel_1th_pre_req = 잔량)
for i in range(1, 11):
p = abs(to_int(d.get(f"sel_{i}th_pre_bid", "0")))
q = to_int(d.get(f"sel_{i}th_pre_req", "0"))
if p > 0:
ask.append({"price": p, "qty": q})
# 매수 호가 1~10
for i in range(1, 11):
p = abs(to_int(d.get(f"buy_{i}th_pre_bid", "0")))
q = to_int(d.get(f"buy_{i}th_pre_req", "0"))
if p > 0:
bid.append({"price": p, "qty": q})
return {
"code": code,
"ask": ask, # 매도 (가격 낮은 것부터)
"bid": bid, # 매수 (가격 높은 것부터)
"ask_total": to_int(d.get("tot_sel_req", "0")),
"bid_total": to_int(d.get("tot_buy_req", "0")),
"base_time": d.get("bid_req_base_tm", ""),
"timestamp": datetime.now().isoformat(),
}
except Exception as e:
logger.debug("ka10004.err", code=code, error=str(e))
return {}
async def fetch_volume_surge() -> list:
"""ka10023: 거래량급증 종목. path=/api/dostk/rkinfo"""
try:
async with httpx.AsyncClient() as c:
d = await kiwoom_post(c, "/api/dostk/rkinfo", "ka10023",
{"mrkt_tp": "000", # 000=전체, 001=코스피, 101=코스닥
"sort_tp": "1", # 1=급증량, 2=급증률
"tm_tp": "2", # 1=분, 2=전일
"trde_qty_tp": "5", # 1=5천주, 2=1만주, 5=5만주
"tm": "",
"stk_cnd": "0", # 0=전체
"pric_tp": "0", # 0=전체
"stex_tp": "3"}) # 3=통합
if d.get("return_code", -1) != 0:
return []
rows = d.get("trde_qty_sdnin", []) or []
result = []
for r in rows[:50]:
raw_code = r.get("stk_cd", "")
# NXT 등 접미사 제거 ('_AL', '_NX' 등)
clean_code = raw_code.split("_")[0] if raw_code else ""
result.append({
"code": clean_code,
"name": r.get("stk_nm", ""),
"price": abs(to_int(r.get("cur_prc", "0"))),
"change_pct": to_float(r.get("flu_rt", "0")),
"volume": to_int(r.get("now_trde_qty", r.get("trde_qty", "0"))),
"prev_volume":to_int(r.get("pred_trde_qty", "0")),
"surge_rate": to_float(r.get("sdnin_rt", r.get("sdnin_qty", "0"))),
})
return result
except Exception as e:
logger.debug("ka10023.err", error=str(e))
return []
async def fetch_short_sale(client: httpx.AsyncClient, code: str, sem: asyncio.Semaphore) -> list:
"""ka10014: 공매도 추이"""
async with sem:
try:
today = datetime.now().strftime("%Y%m%d")
month_ago = (datetime.now() - timedelta(days=30)).strftime("%Y%m%d")
d = await kiwoom_post(client, "/api/dostk/shsa", "ka10014",
{"stk_cd": code, "strt_dt": month_ago, "end_dt": today})
rows = d.get("shrts_trnsn", [])
result = []
for r in rows:
result.append({
"dt": r.get("dt", ""),
"close_price": abs(to_int(r.get("close_pric", "0"))),
"short_qty": to_int(r.get("shrts_qty", "0")),
"short_balance_qty": to_int(r.get("ovr_shrts_qty", "0")),
"trade_weight": to_float(r.get("trde_wght", "0")),
"short_avg_price": to_int(r.get("shrts_avg_pric", "0")),
})
return result
except Exception as e:
logger.debug("ka10014.err", code=code, error=str(e))
return []
# ── 종목 코드 로드 ────────────────────────────────────────
async def get_stock_codes(limit: int = 0) -> list:
"""is_active 종목 전체 (limit=0이면 제한 없음)"""
if pg_pool:
try:
if limit > 0:
rows = await pg_pool.fetch(
"SELECT stock_code FROM dart_corps WHERE is_active=TRUE ORDER BY stock_code LIMIT $1", limit)
else:
rows = await pg_pool.fetch(
"SELECT stock_code FROM dart_corps WHERE is_active=TRUE ORDER BY stock_code")
codes = [r["stock_code"] for r in rows if r["stock_code"]]
if len(codes) >= 50:
return codes
except Exception:
pass
return []
# ── 저장 함수 ─────────────────────────────────────────────
async def save_price(info: dict):
if redis_cl:
redis_data = {
"code": info["code"], "name": info["name"],
"price": info["price"], "change_pct": info["change_pct"],
"change_amount": info["change_amount"],
"volume": info["volume"], "high": info["high"], "low": info["low"],
"open_price": info["open_price"],
"market_cap": info["market_cap"],
"per": info["per"], "pbr": info["pbr"],
"eps": info["eps"], "bps": info["bps"],
"roe": info["roe"], "ev": info["ev"],
"high_52w": info["high_52w"], "low_52w": info["low_52w"],
"foreign_ratio": info["foreign_ratio"],
"credit_ratio": info["credit_ratio"],
"timestamp": info["timestamp"],
}
await redis_cl.set(f"price:{info['code']}", json.dumps(redis_data, ensure_ascii=False), ex=600)
if pg_pool:
try:
async with pg_pool.acquire() as c:
await c.execute("""
INSERT INTO stock_prices (
stock_code, stock_name, price, change_pct, change_amount,
volume, high, low, open_price, market_cap,
per, pbr, eps, bps, roe, ev,
high_52w, low_52w, foreign_ratio, credit_ratio, float_shares, collected_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22)
""", info["code"], info["name"], info["price"], info["change_pct"], info["change_amount"],
info["volume"], info["high"], info["low"], info["open_price"], info["market_cap"],
info["per"], info["pbr"], info["eps"], info["bps"], info["roe"], info["ev"],
info["high_52w"], info["low_52w"], info["foreign_ratio"], info["credit_ratio"],
info["float_shares"], datetime.now())
except Exception as e:
logger.debug("save_price.err", code=info["code"], error=str(e))
async def save_ohlcv(code: str, rows: list):
if not rows or not pg_pool:
return
# Redis에 최근 60일 일봉 저장 (ta-engine 보조)
if redis_cl:
await redis_cl.set(f"ohlcv:{code}", json.dumps(rows[:60], ensure_ascii=False), ex=86400)
async with pg_pool.acquire() as c:
for r in rows:
try:
dt = datetime.strptime(r["dt"], "%Y%m%d").date()
await c.execute("""
INSERT INTO stock_ohlcv (
stock_code, dt, open_price, high_price, low_price, close_price,
volume, trade_amount, foreign_ratio, foreign_net, institution_net, individual_net)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)
ON CONFLICT (stock_code, dt) DO UPDATE SET
close_price=$6, volume=$7, foreign_ratio=$9,
foreign_net=$10, institution_net=$11, individual_net=$12
""", code, dt, r["open"], r["high"], r["low"], r["close"],
r["volume"], r["trade_amount"], r["foreign_ratio"],
r["foreign_net"], r["institution_net"], r["individual_net"])
except Exception:
pass
async def save_foreign_flow(code: str, rows: list):
if not rows or not pg_pool:
return
if redis_cl and rows:
await redis_cl.set(f"foreign:{code}", json.dumps(rows[:20], ensure_ascii=False), ex=86400)
async with pg_pool.acquire() as c:
for r in rows:
try:
dt = datetime.strptime(r["dt"], "%Y%m%d").date()
await c.execute("""
INSERT INTO stock_foreign_flow (
stock_code, dt, close_price, change_qty, hold_qty, hold_ratio, limit_ratio)
VALUES ($1,$2,$3,$4,$5,$6,$7)
ON CONFLICT (stock_code, dt) DO UPDATE SET
change_qty=$4, hold_qty=$5, hold_ratio=$6, limit_ratio=$7
""", code, dt, r["close_price"], r["change_qty"],
r["hold_qty"], r["hold_ratio"], r["limit_ratio"])
except Exception:
pass
async def save_short_sale(code: str, rows: list):
if not rows or not pg_pool:
return
if redis_cl and rows:
await redis_cl.set(f"short:{code}", json.dumps(rows[:20], ensure_ascii=False), ex=86400)
async with pg_pool.acquire() as c:
for r in rows:
try:
dt = datetime.strptime(r["dt"], "%Y%m%d").date()
await c.execute("""
INSERT INTO stock_short_sale (
stock_code, dt, close_price, short_qty, short_balance_qty,
trade_weight, short_avg_price)
VALUES ($1,$2,$3,$4,$5,$6,$7)
ON CONFLICT (stock_code, dt) DO UPDATE SET
short_qty=$4, short_balance_qty=$5, trade_weight=$6, short_avg_price=$7
""", code, dt, r["close_price"], r["short_qty"],
r["short_balance_qty"], r["trade_weight"], r["short_avg_price"])
except Exception:
pass
# ── 수집 작업 ─────────────────────────────────────────────
class Stats:
collected = 0; errors = 0; last_run = ""; last_full_run = ""
stats = Stats()
async def job_price():
"""평일 9~16시 5분마다: 현재가·재무지표 수집 (ka10001)"""
codes = await get_stock_codes(0)
if not codes:
logger.warning("job_price.no_codes")
return
sem = asyncio.Semaphore(10) # 동시 10개 제한
ok = 0
async with httpx.AsyncClient() as client:
tasks = [fetch_basic_info(client, code, sem) for code in codes]
results = await asyncio.gather(*tasks, return_exceptions=True)
for info in results:
if isinstance(info, dict) and info.get("price", 0) > 0:
await save_price(info)
ok += 1
else:
stats.errors += 1
await gen_signals()
stats.collected += ok
stats.last_run = datetime.now().isoformat()
if redis_cl:
await redis_cl.set("prices:last_update", datetime.now().isoformat())
logger.info("job_price.done", ok=ok, total=len(codes))
async def job_full(days: int = 10):
"""평일 17:00: 일봉·외국인·공매도 전체 수집 (ka10005·ka10008·ka10014).
일별은 days=10(경량, ON CONFLICT 갱신). 백필은 /collect/full?days=400."""
codes = await get_stock_codes(0)
if not codes:
return
sem = asyncio.Semaphore(5) # 야간 수집 느리게
ok = 0
async with httpx.AsyncClient() as client:
for code in codes:
try:
ohlcv, foreign, short = await asyncio.gather(
fetch_ohlcv(client, code, sem, days),
fetch_foreign_flow(client, code, sem),
fetch_short_sale(client, code, sem),
return_exceptions=True
)
if isinstance(ohlcv, list):
await save_ohlcv(code, ohlcv)
if isinstance(foreign, list):
await save_foreign_flow(code, foreign)
if isinstance(short, list):
await save_short_sale(code, short)
ok += 1
except Exception as e:
logger.debug("job_full.err", code=code, error=str(e))
stats.last_full_run = datetime.now().isoformat()
logger.info("job_full.done", ok=ok, total=len(codes))
# ── 시그널 생성 ───────────────────────────────────────────
def calc_signal(info: dict, news_sc: float, dart_sc: float,
foreign_net_5d: int, short_weight: float) -> Optional[dict]:
price = info["price"]
if price <= 0:
return None
h52 = info.get("high_52w", 0)
l52 = info.get("low_52w", 0)
cpct = info.get("change_pct", 0)
foreign_ratio = info.get("foreign_ratio", 0)
# 외국인 수급 보너스 (5일 누적 순매수 양수면 +10, 음수면 -10)
foreign_bonus = min(10, max(-10, foreign_net_5d / 500_000)) if foreign_net_5d else 0
# 공매도 패널티 (비중 5% 이상이면 -5)
short_penalty = -5 if short_weight >= 5 else 0
total = (news_sc * 0.35 + dart_sc * 0.25
+ (cpct * 10) * 0.25 + foreign_bonus * 0.10 + short_penalty * 0.05)
total = max(-100, min(100, total))
pos = (price - l52) / (h52 - l52) if h52 > l52 > 0 else 0.5
if total >= 30:
sig = "매수"
tp = int(price + (h52 - price) * (0.3 + min(total, 100) / 200)) if h52 > price else int(price * 1.10)
sl = int(price * (0.92 + pos * 0.03))
conf = min(95, 50 + total * 0.3 + (1 - pos) * 20)
elif total <= -30:
sig = "매도"
tp = int(price - (price - l52) * (0.3 + min(abs(total), 100) / 200)) if l52 > 0 and l52 < price else int(price * 0.90)
sl = int(price * 1.05)
conf = min(95, 50 + abs(total) * 0.3 + pos * 20)
else:
return None
er = ((tp - price) / price) * 100 if sig == "매수" else ((price - tp) / price) * 100
risk = abs(sl - price)
reward = abs(tp - price)
rr = reward / risk if risk > 0 else 0
return {
"signal_type": sig, "current_price": price, "target_price": tp, "stop_loss": sl,
"expected_return_pct": round(er, 2), "risk_reward_ratio": round(rr, 2),
"confidence": round(conf, 1), "price_momentum": round(cpct, 2),
"news_score": round(news_sc, 1), "dart_score": round(dart_sc, 1),
"foreign_net_5d": foreign_net_5d, "short_weight": short_weight,
}
async def gen_signals():
if not pg_pool:
return
async with pg_pool.acquire() as c:
scores = await c.fetch("""
SELECT stock_code, stock_name, news_score, dart_score, total_score
FROM stock_scores
WHERE score_date = (SELECT MAX(score_date) FROM stock_scores)
AND (total_score >= 30 OR total_score <= -30)
""")
n = 0
for row in scores:
code = row["stock_code"]
if not code or len(code) != 6:
continue
if not redis_cl:
continue
cached = await redis_cl.get(f"price:{code}")
if not cached:
continue
info = json.loads(cached)
# 외국인 5일 순매수 합산
foreign_net_5d = 0
f_cached = await redis_cl.get(f"foreign:{code}")
if f_cached:
f_rows = json.loads(f_cached)
foreign_net_5d = sum(r.get("change_qty", 0) for r in f_rows[:5])
# 공매도 최근 비중
short_weight = 0.0
s_cached = await redis_cl.get(f"short:{code}")
if s_cached:
s_rows = json.loads(s_cached)
if s_rows:
short_weight = s_rows[0].get("trade_weight", 0.0)
sig = calc_signal(info, row["news_score"], row["dart_score"],
foreign_net_5d, short_weight)
if not sig:
continue
reasons = await c.fetch("""
SELECT reason FROM news_analysis
WHERE primary_stock=$1 AND intensity>=2
ORDER BY analyzed_at DESC LIMIT 2
""", code)
reason = " | ".join([r["reason"][:100] for r in reasons])
await c.execute("""
INSERT INTO trade_signals (
stock_code, stock_name, signal_type, current_price,
target_price, stop_loss, expected_return_pct, risk_reward_ratio,
confidence, reason, news_score, dart_score, price_momentum,
foreign_net_5d, short_weight)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15)
ON CONFLICT (stock_code, (created_at::date)) DO UPDATE SET
signal_type=EXCLUDED.signal_type, current_price=EXCLUDED.current_price,
target_price=EXCLUDED.target_price, stop_loss=EXCLUDED.stop_loss,
expected_return_pct=EXCLUDED.expected_return_pct,
risk_reward_ratio=EXCLUDED.risk_reward_ratio,
confidence=EXCLUDED.confidence, reason=EXCLUDED.reason,
news_score=EXCLUDED.news_score, dart_score=EXCLUDED.dart_score,
price_momentum=EXCLUDED.price_momentum,
foreign_net_5d=EXCLUDED.foreign_net_5d, short_weight=EXCLUDED.short_weight
""", code, row["stock_name"], sig["signal_type"], sig["current_price"],
sig["target_price"], sig["stop_loss"], sig["expected_return_pct"],
sig["risk_reward_ratio"], sig["confidence"], reason,
sig["news_score"], sig["dart_score"], sig["price_momentum"],
sig["foreign_net_5d"], sig["short_weight"])
n += 1
logger.info("signals.done", count=n)
# ── FastAPI ───────────────────────────────────────────────
app = FastAPI(title="키움증권 주가·수급·공매도")
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
@app.on_event("startup")
async def startup():
global pg_pool, redis_cl
pg_pool = await asyncpg.create_pool(
host=PG_HOST, port=PG_PORT, database=PG_DB,
user=PG_USER, password=PG_PASS, min_size=2, max_size=5)
redis_cl = aioredis.Redis(
host=REDIS_HOST, port=6379, password=REDIS_PASSWORD, db=3, decode_responses=True)
await init_db()
# 현재가: 평일 9~16시 5분마다
scheduler.add_job(job_price, "cron", day_of_week="mon-fri",
hour="9-15", minute="*/5", id="price", replace_existing=True)
# 장 마감 후 전체 수집: 평일 17:00
scheduler.add_job(job_full, "cron", day_of_week="mon-fri",
hour="17", minute="0", id="full", replace_existing=True)
scheduler.start()
logger.info("kiwoom-api.started")
@app.on_event("shutdown")
async def shutdown():
scheduler.shutdown()
if pg_pool:
await pg_pool.close()
if redis_cl:
await redis_cl.aclose()
@app.get("/health")
async def health():
lu = await redis_cl.get("prices:last_update") if redis_cl else ""
return JSONResponse(content={
"status": "ok", "collected": stats.collected, "errors": stats.errors,
"last_run": stats.last_run, "last_full_run": stats.last_full_run,
"last_update": lu or ""
})
@app.get("/price/{code}")
async def price(code: str):
if redis_cl:
c = await redis_cl.get(f"price:{code}")
if c:
return JSONResponse(content=json.loads(c))
# 실시간 단건 조회
sem = asyncio.Semaphore(1)
async with httpx.AsyncClient() as client:
info = await fetch_basic_info(client, code, sem)
if info:
await save_price(info)
return JSONResponse(content=info)
return JSONResponse(content={"error": "not found"}, status_code=404)
@app.get("/prices")
async def prices(limit: int = Query(default=50)):
if not redis_cl:
return JSONResponse(content={"data": {}})
keys = await redis_cl.keys("price:*")
result = {}
for k in keys[:limit]:
c = await redis_cl.get(k)
if c:
d = json.loads(c)
result[d["code"]] = d
return JSONResponse(content={"count": len(result), "data": result})
@app.get("/foreign/{code}")
async def foreign(code: str):
if redis_cl:
c = await redis_cl.get(f"foreign:{code}")
if c:
return JSONResponse(content={"code": code, "data": json.loads(c)})
sem = asyncio.Semaphore(1)
async with httpx.AsyncClient() as client:
rows = await fetch_foreign_flow(client, code, sem)
await save_foreign_flow(code, rows)
return JSONResponse(content={"code": code, "data": rows})
@app.get("/short/{code}")
async def short_sale(code: str):
if redis_cl:
c = await redis_cl.get(f"short:{code}")
if c:
return JSONResponse(content={"code": code, "data": json.loads(c)})
sem = asyncio.Semaphore(1)
async with httpx.AsyncClient() as client:
rows = await fetch_short_sale(client, code, sem)
await save_short_sale(code, rows)
return JSONResponse(content={"code": code, "data": rows})
@app.get("/ohlcv/{code}")
async def ohlcv(code: str, days: int = Query(default=60)):
if redis_cl:
c = await redis_cl.get(f"ohlcv:{code}")
if c:
data = json.loads(c)
return JSONResponse(content={"code": code, "data": data[:days]})
sem = asyncio.Semaphore(1)
async with httpx.AsyncClient() as client:
rows = await fetch_ohlcv(client, code, sem, days)
await save_ohlcv(code, rows)
return JSONResponse(content={"code": code, "data": rows})
@app.get("/signals")
async def signals(days: int = Query(default=7)):
async with pg_pool.acquire() as c:
rows = await c.fetch("""
SELECT * FROM trade_signals
WHERE created_at >= NOW() - INTERVAL '%s days'
ORDER BY confidence DESC LIMIT 50
""" % days)
return [dict(r) for r in rows]
@app.get("/signals/{code}")
async def stock_signals(code: str):
async with pg_pool.acquire() as c:
rows = await c.fetch("""
SELECT * FROM trade_signals WHERE stock_code=$1
ORDER BY created_at DESC LIMIT 10
""", code)
return [dict(r) for r in rows]
@app.get("/summary/{code}")
async def summary(code: str):
"""종목 종합 요약 (현재가 + 외국인 + 공매도 + 최근 신호)"""
result = {"code": code}
if redis_cl:
for key in [f"price:{code}", f"foreign:{code}", f"short:{code}", f"ohlcv:{code}"]:
c = await redis_cl.get(key)
if c:
field = key.split(":")[0]
result[field] = json.loads(c) if field != "price" else json.loads(c)
return JSONResponse(content=result)
@app.post("/collect/price")
async def collect_price():
asyncio.create_task(job_price())
return {"status": "started", "job": "price"}
@app.post("/collect/full")
async def collect_full(days: int = Query(default=10, ge=1, le=600)):
asyncio.create_task(job_full(days))
return {"status": "started", "job": "full", "days": days}
# ── 추가: 분봉 / 호가 / 거래량급증 ─────────────────────────
@app.get("/minute/{code}")
async def minute_chart(code: str, scope: str = Query(default="5", description="분봉 단위: 1,3,5,10,15,30,60")):
"""ka10080: 분봉차트 (실시간 호출)"""
async with httpx.AsyncClient() as c:
data = await fetch_minute_chart(c, code, scope)
return JSONResponse(content={"code": code, "scope": scope, "data": data})
@app.get("/orderbook/{code}")
async def orderbook(code: str):
"""ka10004: 10단계 호가 잔량"""
async with httpx.AsyncClient() as c:
data = await fetch_orderbook(c, code)
return JSONResponse(content=data or {"code": code, "ask": [], "bid": []})
@app.get("/volume-surge")
async def volume_surge():
"""ka10023: 거래량 급증 종목 TOP50 (Redis 5분 캐시)"""
cached = None
if redis_cl:
try:
v = await redis_cl.get("vol_surge:list")
if v: cached = json.loads(v)
except: pass
if cached:
return JSONResponse(content={"data": cached, "cached": True})
data = await fetch_volume_surge()
if redis_cl and data:
try: await redis_cl.setex("vol_surge:list", 300, json.dumps(data))
except: pass
return JSONResponse(content={"data": data, "cached": False})
+8
View File
@@ -0,0 +1,8 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
httpx==0.27.0
redis==5.0.4
asyncpg==0.29.0
apscheduler==3.10.4
orjson==3.10.3
structlog==24.2.0
+269
View File
@@ -0,0 +1,269 @@
"""
네이버 금융 뉴스 수집기 + AI 분석 파이프라인
- 시장 전체 뉴스 (5분마다)
- 시총 상위 200 종목별 뉴스 (30분마다)
- 수집 즉시 바른API Ollama Qdrant vLLM PostgreSQL
"""
import asyncio, hashlib, json, os, re, random, time
from datetime import datetime, timedelta
from typing import Optional
import asyncpg, httpx, redis.asyncio as aioredis, structlog
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from bs4 import BeautifulSoup
from fastapi import FastAPI, Query
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
structlog.configure(processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(),
])
logger = structlog.get_logger()
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
PG_HOST = os.getenv("POSTGRES_HOST", "postgres")
PG_PORT = int(os.getenv("POSTGRES_PORT", "5432"))
PG_DB = os.getenv("POSTGRES_DB", "trading_ai")
PG_USER = os.getenv("POSTGRES_USER", "kyu")
PG_PASS = os.getenv("POSTGRES_PASSWORD", "7895123")
BAREUN_URL = os.getenv("BAREUN_API_URL", "http://bareunaapi:5757")
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://ollama:11434")
VLLM_URL = os.getenv("VLLM_URL", "http://vllm:8000")
QDRANT_URL = os.getenv("QDRANT_URL", "http://qdrant:6333")
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
pg_pool: Optional[asyncpg.Pool] = None
redis_cl: Optional[aioredis.Redis] = None
scheduler = AsyncIOScheduler(timezone="Asia/Seoul")
class S:
collected = 0; processed = 0; duplicates = 0; errors = 0
last_run = ""; running = False
stats = S()
def nhash(title, url=""): return hashlib.sha256(f"{title.strip()}{url.strip()}".encode()).hexdigest()[:16]
async def is_dup(h):
if not redis_cl: return False
try:
r = await redis_cl.set(f"news:naver:{h}", "1", ex=86400, nx=True)
return r is None
except: return False
# ── 크롤러 ────────────────────────────────────────────────
async def crawl_market_news(client):
news = []
urls = [
"https://finance.naver.com/news/mainnews.naver",
"https://finance.naver.com/news/news_list.naver?mode=LSS2D&section_id=101&section_id2=258",
"https://finance.naver.com/news/news_list.naver?mode=LSS2D&section_id=101&section_id2=259",
"https://finance.naver.com/news/news_list.naver?mode=LSS2D&section_id=101&section_id2=261",
]
for url in urls:
try:
r = await client.get(url, headers=HEADERS, timeout=15)
r.encoding = "euc-kr"
soup = BeautifulSoup(r.text, "lxml")
for a in soup.select("a[href*='article_id']"):
t = a.get_text(strip=True)
h = a.get("href", "")
if t and len(t) > 10:
news.append({"title": t, "url": f"https://finance.naver.com{h}" if h.startswith("/") else h,
"source": "네이버금융", "content": "", "published_at": datetime.now().isoformat()})
await asyncio.sleep(0.3)
except Exception as e:
logger.warning("crawl.market.err", error=str(e))
return news
async def crawl_stock_news(client, code, name):
news = []
try:
r = await client.get(f"https://finance.naver.com/item/news_news.naver?code={code}&page=1", headers=HEADERS, timeout=15)
r.encoding = "euc-kr"
soup = BeautifulSoup(r.text, "lxml")
for tr in soup.select("table.type5 tr"):
a = tr.select_one("td.title a")
dt = tr.select_one("td.date")
src = tr.select_one("td.info")
if a:
t = a.get_text(strip=True)
if t and len(t) > 5:
news.append({"title": t, "url": f"https://finance.naver.com{a.get('href','')}",
"source": src.get_text(strip=True) if src else "네이버금융",
"content": f"[{name}({code})] {t}",
"published_at": dt.get_text(strip=True) if dt else datetime.now().isoformat(),
"stock_code": code, "stock_name": name})
except: pass
return news
async def get_top_stocks(client, count=200):
stocks = []
for sosok in [0, 1]:
for page in range(1, 50):
try:
r = await client.get(f"https://finance.naver.com/sise/sise_market_sum.naver?sosok={sosok}&page={page}", headers=HEADERS, timeout=15)
r.encoding = "euc-kr"
rows = re.findall(r'main\.naver\?code=(\d{6})[^>]*>([^<]+)</a>', r.text)
if not rows: break
for c, n in rows: stocks.append({"code": c.strip(), "name": n.strip()})
await asyncio.sleep(0.2)
except: break
if len(stocks) >= count: break
return stocks[:count]
# ── 파이프라인 ─────────────────────────────────────────────
async def pipeline(item, client):
try:
# 1. 바른API
br = await client.post(f"{BAREUN_URL}/analyze", json={
"title": item["title"], "content": item.get("content",""),
"url": item.get("url",""), "source": item.get("source",""),
"published_at": item.get("published_at","")}, timeout=30)
bd = br.json()
if bd.get("is_duplicate"): return "dup"
data = {**item, "hash": bd.get("hash",""), "stocks": bd.get("stocks",[]),
"keywords": bd.get("keywords",[]), "filtered_text": bd.get("filtered_text","")}
# 2. Ollama 임베딩
er = await client.post(f"{OLLAMA_URL}/api/embeddings",
json={"model":"bge-m3","prompt": data["filtered_text"] or data["title"]}, timeout=60)
emb = er.json().get("embedding")
if not emb: return "no_embed"
# 3. Qdrant 유사도
try:
sr = await client.post(f"{QDRANT_URL}/collections/news_vectors/points/search",
json={"vector":emb,"limit":3,"score_threshold":0.85,"with_payload":True}, timeout=15)
hits = sr.json().get("result",[])
if any(h["score"]>=0.92 and h["score"]<0.99 for h in hits): return "sim_dup"
except: hits = []
# 4. vLLM 분석
stocks_str = ", ".join([f'{s["name"]}({s["code"]})' for s in data["stocks"][:5]])
prompt = (f"뉴스: {data['title'][:200]}\n키워드: {(data['filtered_text'] or '')[:300]}\n"
f"감지된 종목: {stocks_str}\n\n"
"JSON 응답. primary_stock에 6자리 종목코드, 시장전체면 KOSPI/KOSDAQ:\n"
'{"sentiment":"호재/악재/중립","intensity":1~5,"primary_stock":"종목코드",'
'"affected_stocks":["코드"],"reason":"근거","investment_action":"매수관심/매도관심/관망"}')
vr = await client.post(f"{VLLM_URL}/v1/chat/completions", json={
"model":"exaone","messages":[
{"role":"system","content":"한국 주식 전문 애널리스트. JSON만 응답."},
{"role":"user","content":prompt}],
"max_tokens":300,"temperature":0.1}, timeout=120)
try:
c = vr.json()["choices"][0]["message"]["content"]
a = json.loads(re.sub(r"```json\n?|```","",c).strip())
except:
a = {"sentiment":"중립","intensity":0,"primary_stock":"","affected_stocks":[],"reason":"파싱실패","investment_action":"관망"}
# 5. Qdrant 저장
try:
await client.put(f"{QDRANT_URL}/collections/news_vectors/points", json={
"points":[{"id":random.randint(1,999999999),"vector":emb,
"payload":{"title":data["title"],"hash":data["hash"],
"sentiment":a.get("sentiment",""),"intensity":a.get("intensity",0),
"primary_stock":a.get("primary_stock","")}}]}, timeout=15)
except: pass
# 6. PostgreSQL 저장
esc = lambda s: (s or "").replace("'","''")
await pg_pool.execute(f"""
INSERT INTO news_analysis (title,url,source,published_at,hash,sentiment,intensity,
primary_stock,affected_stocks,reason,investment_action,keywords,stock_names,stock_codes,similar_count,analyzed_at)
VALUES ('{esc(data["title"][:500])}','{esc(data.get("url","")[:500])}','{esc(data.get("source","")[:100])}',
'{data.get("published_at",datetime.now().isoformat())}','{data["hash"]}',
'{a.get("sentiment","중립")}',{a.get("intensity",0)},'{esc(a.get("primary_stock",""))}',
'{json.dumps(a.get("affected_stocks",[]))}','{esc(a.get("reason","")[:500])}',
'{a.get("investment_action","관망")}','{json.dumps(data.get("keywords",[])[:20])}',
'{json.dumps([s["name"] for s in data.get("stocks",[])])}',
'{json.dumps([s["code"] for s in data.get("stocks",[])])}',
0,'{datetime.now().isoformat()}')
ON CONFLICT (hash) DO NOTHING""")
return "ok"
except Exception as e:
logger.warning("pipeline.err", title=item.get("title","")[:50], error=str(e))
return "error"
# ── 수집 작업 ──────────────────────────────────────────────
async def job_market():
if stats.running: return
stats.running = True
try:
async with httpx.AsyncClient() as c:
news = await crawl_market_news(c)
ok = 0
for item in news:
h = nhash(item["title"], item.get("url",""))
if await is_dup(h): stats.duplicates+=1; continue
stats.collected += 1
r = await pipeline(item, c)
if r == "ok": ok += 1; stats.processed += 1
await asyncio.sleep(0.5)
stats.last_run = datetime.now().isoformat()
logger.info("job.market", total=len(news), processed=ok)
except Exception as e:
stats.errors += 1; logger.error("job.market.err", error=str(e))
finally:
stats.running = False
async def job_stocks():
if stats.running: return
stats.running = True
try:
async with httpx.AsyncClient() as c:
top = await get_top_stocks(c, 200)
ok = 0
for stock in top:
try:
news = await crawl_stock_news(c, stock["code"], stock["name"])
for item in news:
h = nhash(item["title"], item.get("url",""))
if await is_dup(h): stats.duplicates+=1; continue
stats.collected += 1
r = await pipeline(item, c)
if r == "ok": ok += 1; stats.processed += 1
await asyncio.sleep(0.3)
except: pass
stats.last_run = datetime.now().isoformat()
logger.info("job.stocks", stocks=len(top), processed=ok)
except Exception as e:
stats.errors += 1; logger.error("job.stocks.err", error=str(e))
finally:
stats.running = False
# ── FastAPI ────────────────────────────────────────────────
app = FastAPI(title="뉴스 수집기")
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
@app.on_event("startup")
async def startup():
global pg_pool, redis_cl
pg_pool = await asyncpg.create_pool(host=PG_HOST,port=PG_PORT,database=PG_DB,user=PG_USER,password=PG_PASS,min_size=2,max_size=5)
redis_cl = aioredis.Redis(host=REDIS_HOST,port=6379,password=REDIS_PASSWORD,db=4,decode_responses=True)
scheduler.add_job(job_market,"cron",day_of_week="mon-fri",hour="8-18",minute="*/5",id="market",replace_existing=True)
scheduler.add_job(job_stocks,"cron",day_of_week="mon-fri",hour="9-16",minute="*/30",id="stocks",replace_existing=True)
scheduler.start()
logger.info("news-collector.started")
@app.on_event("shutdown")
async def shutdown():
scheduler.shutdown()
if pg_pool: await pg_pool.close()
if redis_cl: await redis_cl.aclose()
@app.get("/health")
async def health():
return JSONResponse(content={"status":"ok","collected":stats.collected,"processed":stats.processed,
"duplicates":stats.duplicates,"errors":stats.errors,"last_run":stats.last_run,"running":stats.running})
@app.post("/collect/market")
async def m_market():
asyncio.create_task(job_market()); return {"status":"started","type":"market"}
@app.post("/collect/stocks")
async def m_stocks():
asyncio.create_task(job_stocks()); return {"status":"started","type":"stocks"}
+8
View File
@@ -0,0 +1,8 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8787
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8787", "--workers", "1", "--log-level", "info"]
+886
View File
@@ -0,0 +1,886 @@
"""
멀티소스 금융 뉴스 수집기 + AI 분석 파이프라인 (버핏 스타일 강화)
- RSS 28 소스 (네이버 크롤링 제거, 주식 전문 사이트 중심)
- 5분마다 전체 소스 수집
- 2단계 중복제거: URL해시(Redis) + 제목정규화해시(Redis) + 벡터유사도(Qdrant)
- 수집 즉시 바른API Ollama Qdrant Ollama(EXAONE) PostgreSQL
"""
import asyncio, hashlib, json, os, re, random, time
import xml.etree.ElementTree as ET
from datetime import datetime, timedelta
from typing import Optional
import asyncpg, httpx, redis.asyncio as aioredis, structlog
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from bs4 import BeautifulSoup
from fastapi import FastAPI, Query
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
structlog.configure(processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(),
])
logger = structlog.get_logger()
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
PG_HOST = os.getenv("POSTGRES_HOST", "postgres")
PG_PORT = int(os.getenv("POSTGRES_PORT", "5432"))
PG_DB = os.getenv("POSTGRES_DB", "trading_ai")
PG_USER = os.getenv("POSTGRES_USER", "kyu")
PG_PASS = os.getenv("POSTGRES_PASSWORD", "7895123")
BAREUN_URL = os.getenv("BAREUN_API_URL", "http://bareunaapi:5757")
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://ollama:11434")
QDRANT_URL = os.getenv("QDRANT_URL", "http://qdrant:6333")
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
pg_pool: Optional[asyncpg.Pool] = None
redis_cl: Optional[aioredis.Redis] = None
scheduler = AsyncIOScheduler(timezone="Asia/Seoul")
class S:
collected = 0; processed = 0; duplicates = 0; errors = 0
noise = 0; off_topic = 0
last_run = ""; running = False
stats = S()
def nhash(title, url=""): return hashlib.sha256(f"{title.strip()}{url.strip()}".encode()).hexdigest()[:16]
def normalize_title(title: str) -> str:
"""[속보][단독](종합) 등 제거 후 특수문자·공백 제거 → 유사 제목 중복 감지용"""
t = re.sub(r'[\[\(【〔][^\]\)】〕]{0,10}[\]\)】〕]', '', title)
t = re.sub(r'[^\w가-힣a-zA-Z0-9]', '', t)
return t.lower().strip()
def is_korean(text: str) -> bool:
if not text: return True
hangul = sum(1 for c in text if '' <= c <= '')
cjk = sum(1 for c in text if '' <= c <= '鿿' or '' <= c <= '䶿')
if hangul == 0 and cjk > 2: return False
return True
async def is_dup(title: str, url: str = "") -> bool:
if not redis_cl: return False
try:
# 1차: URL+제목 해시 (완전 동일 기사)
h_url = nhash(title, url)
if await redis_cl.set(f"news:u:{h_url}", "1", ex=172800, nx=True) is None:
return True
# 2차: 정규화된 제목 해시 (같은 뉴스 다른 소스)
h_norm = nhash(normalize_title(title))
if await redis_cl.set(f"news:t:{h_norm}", "1", ex=21600, nx=True) is None:
return True
return False
except: return False
# ── RSS 멀티소스 수집 (28개 소스, 네이버 크롤링 제거) ──────
RSS_SOURCES = [
# ── 주요 경제/금융 신문 ──
("한국경제", "https://www.hankyung.com/feed/all-news"),
("한국경제", "https://www.hankyung.com/feed/finance"),
("매일경제", "https://www.mk.co.kr/rss/30100041/"),
("매일경제", "https://www.mk.co.kr/rss/30200030/"),
("머니투데이", "https://news.mt.co.kr/mtview.php?type=rss&MTPub=E&section=E"),
("머니투데이", "https://news.mt.co.kr/mtview.php?type=rss&MTPub=S"),
("이데일리", "https://www.edaily.co.kr/rss/news/finance.xml"),
("이데일리", "https://www.edaily.co.kr/rss/news/markets.xml"),
("연합뉴스", "https://www.yna.co.kr/rss/economy.xml"),
("연합뉴스", "https://www.yna.co.kr/rss/market.xml"),
("조선비즈", "https://biz.chosun.com/rss/rss.htm?site=biz.chosun.com"),
("헤럴드경제", "https://biz.heraldcorp.com/rss/index.xml"),
("파이낸셜뉴스", "https://www.fnnews.com/rss/fn_economy.xml"),
("서울경제", "https://www.sedaily.com/RSS/Economy"),
("아시아경제", "https://www.asiae.co.kr/rss/economy.htm"),
("뉴스1", "https://www.news1.kr/rss/industry.xml"),
("비즈니스포스트", "https://www.businesspost.co.kr/BP?command=rss&rssFeedType=0000"),
("인포스탁", "https://www.infostock.co.kr/rss/rss_news.xml"),
# ── 추가 소스 ──
("뉴스핌", "https://www.newspim.com/rss/view?outtype=1"),
("뉴시스", "https://www.newsis.com/rss/finance.xml"),
("이투데이", "https://www.etoday.co.kr/news/newsfeed.php?CateId=0102"),
("글로벌이코노믹", "https://www.g-enews.com/rss/rss_economy.xml"),
("전자신문", "https://www.etnews.com/news/latest_news.xml"),
("디지털타임스", "https://www.dt.co.kr/rss/rss_economy.html"),
("더벨", "https://www.thebell.co.kr/free/content/xmlService.asp"),
("스탁데일리", "https://www.stockdaily.kr/rss/rss.xml"),
("세계일보", "https://www.segye.com/RSS/economyRss.xml"),
("SBS Biz", "https://news.sbs.co.kr/news/SectionRssFeed.do?sectionId=EC"),
]
def parse_rss_date(date_str: str) -> str:
if not date_str:
return datetime.now().isoformat()
for fmt in ("%a, %d %b %Y %H:%M:%S %z", "%a, %d %b %Y %H:%M:%S +0900",
"%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%d %H:%M:%S"):
try:
return datetime.strptime(date_str.strip(), fmt).isoformat()
except: pass
return datetime.now().isoformat()
async def crawl_rss_sources(client) -> list:
news = []
for source_name, rss_url in RSS_SOURCES:
try:
r = await client.get(rss_url, timeout=15,
headers={"User-Agent": "Mozilla/5.0", "Accept": "application/rss+xml, application/xml"})
if r.status_code != 200:
continue
try:
root = ET.fromstring(r.content)
except ET.ParseError:
# XML 파싱 실패 시 BeautifulSoup 폴백
soup = BeautifulSoup(r.content, "lxml-xml")
items = soup.find_all("item")
for item in items[:20]:
title = item.find("title")
link = item.find("link")
pub = item.find("pubDate") or item.find("dc:date")
if title and title.get_text(strip=True):
t = title.get_text(strip=True)
if len(t) > 5 and is_korean(t):
news.append({
"title": t,
"url": link.get_text(strip=True) if link else "",
"source": source_name,
"content": "",
"published_at": parse_rss_date(pub.get_text(strip=True) if pub else ""),
})
continue
ns = {"dc": "http://purl.org/dc/elements/1.1/"}
items = root.findall(".//item")
for item in items[:20]:
title = item.findtext("title", "").strip()
link = item.findtext("link", "").strip()
pubdate = item.findtext("pubDate", "") or item.findtext("dc:date", "", ns)
desc = item.findtext("description", "")
if title and len(title) > 5 and is_korean(title):
news.append({
"title": title,
"url": link,
"source": source_name,
"content": re.sub(r"<[^>]+>", "", desc)[:300] if desc else "",
"published_at": parse_rss_date(pubdate),
})
await asyncio.sleep(0.2)
except Exception as e:
logger.debug("rss.fetch.err", source=source_name, error=str(e))
logger.info("rss.crawled", sources=len(RSS_SOURCES), items=len(news))
return news
# ── 비주식 뉴스 필터 ───────────────────────────────────────
# 명백한 비주식 카테고리 prefix → 수집 차단
NOISE_PREFIXES = (
# 도서/문화
"[책마을]", "[책꽂이]", "[책&생각]", "[책]", "[리뷰]", "[영화]",
"[공연]", "[전시]", "[음악]", "[연예]",
# 의견/사설
"[사설]", "[칼럼]", "[기고]", "[오피니언]", "[기자수첩]", "[데스크칼럼]",
"[Why]", "[취재일기]", "[현장에서]", "[광화문]", "[로터리]",
# 인사/사회
"[부고]", "[인사]", "[동정]", "[포토]", "[화보]", "[지면]", "[그래픽]",
# 라이프
"[건강]", "[푸드]", "[여행]", "[패션]", "[뷰티]", "[자동차]", "[취미]",
"[운세]", "[날씨]",
# 스포츠
"[스포츠]", "[야구]", "[축구]", "[농구]", "[골프]",
# 광고성/이벤트
"[보도자료]", "[알림]", "[이벤트]", "[알립니다]",
)
# 종목/시장과 무관한 라이프스타일·클릭베이트 키워드 (제목에 등장하면 차단)
NOISE_KEYWORDS = (
# 라이프/소비
"이직", "퇴사", "신의 직장", "마통", "돈복사", "전기료 폭탄",
"세탁기", "건조기", "에어컨", "다이어트", "맛집", "할인",
"결혼", "이혼", "연애", "임신", "출산", "육아",
# 사회/사건사고 (기업이 부수적으로 언급되는 경우 多)
"성폭행", "성추행", "음주운전", "마약", "도박", "사기",
"교통사고", "화재", "살인", "강도",
# 정치 (종목 영향 적은 일반 정치)
"선거", "투표", "여론조사", "지지율", "총선", "대선",
# 라이프스타일 클릭베이트
"충격", "발칵", "경악", "", "화들짝", "깜짝", "대체",
)
# 클릭베이트: 제목이 큰따옴표·작은따옴표로 시작하면 의심
CLICKBAIT_QUOTE_RE = re.compile(r'^[\s]*[\"“‘’\']')
# 종목 매칭 0건일 때 시장 관련성 판정용 키워드
MARKET_KEYWORDS = (
# 지수/시장
"코스피", "코스닥", "KOSPI", "KOSDAQ", "나스닥", "다우", "S&P", "증시",
"지수", "시장", "장세", "장중", "거래소",
# 거시지표
"환율", "금리", "원달러", "원화", "달러", "엔화", "위안", "유가",
"금값", "은값", "원유", "WTI", "브렌트", "채권", "국채", "회사채",
"인플레이션", "물가", "CPI", "PPI", "성장률", "GDP", "경기", "경제",
# 수급
"외국인", "외인", "기관", "개인투자자", "프로그램매매", "공매도",
"신용잔고", "거래대금", "거래량",
# 종목/거래
"주식", "주가", "종목", "매수", "매도", "매매", "투자", "테마주", "관련주",
"급등", "급락", "상승", "하락", "신고가", "신저가", "호재", "악재",
# 펀드/상품
"펀드", "ETF", "ETN", "선물", "옵션", "파생", "리츠",
# 기업이벤트
"상장", "상폐", "공모", "IPO", "유상증자", "무상증자", "배당", "자사주",
"M&A", "인수", "합병", "분할", "지분", "스톡옵션",
# 실적/재무
"실적", "영업이익", "매출", "순이익", "EBITDA", "ROE", "PER", "PBR",
"수주", "계약", "수출", "공시",
# 산업/섹터
"산업", "섹터", "반도체", "바이오", "2차전지", "배터리", "전기차",
"AI", "로봇", "방산", "조선", "건설", "유통", "금융", "은행", "보험",
"증권", "통신", "에너지", "철강", "화학", "정유"
)
def _is_noise_title(title: str) -> bool:
if not title:
return True
if any(title.startswith(p) for p in NOISE_PREFIXES):
return True
# 클릭베이트: 따옴표 시작 + 시장 키워드 부재 시 차단
if CLICKBAIT_QUOTE_RE.match(title) and not any(kw in title for kw in MARKET_KEYWORDS):
return True
# 라이프스타일/사건사고 키워드 (제목에서 검사)
if any(nk in title for nk in NOISE_KEYWORDS):
return True
return False
def _has_market_relevance(text: str) -> bool:
"""종목 매칭이 없을 때만 호출. 시장 키워드 ≥2개 매칭 시 통과 (broad 1개로는 부족)"""
if not text:
return False
matches = sum(1 for kw in MARKET_KEYWORDS if kw in text)
return matches >= 2
# ── 파이프라인 ─────────────────────────────────────────────
_corp_cache: dict = {}
async def _corp_name(code: str) -> str:
if code in _corp_cache:
return _corp_cache[code]
nm = code
try:
row = await pg_pool.fetchrow(
"SELECT corp_name FROM dart_corps WHERE stock_code=$1", code)
if row and row["corp_name"]:
nm = row["corp_name"]
except Exception:
pass
_corp_cache[code] = nm
return nm
async def pipeline(item, client):
try:
# 0. 비주식 카테고리 prefix 차단 (바른API 호출 전)
if _is_noise_title(item.get("title", "")):
return "noise"
# 1. 바른API
br = await client.post(f"{BAREUN_URL}/analyze", json={
"title": item["title"], "content": item.get("content",""),
"url": item.get("url",""), "source": item.get("source",""),
"published_at": item.get("published_at","")}, timeout=30)
bd = br.json()
if bd.get("is_duplicate"): return "dup"
data = {**item, "hash": bd.get("hash",""), "stocks": bd.get("stocks",[]),
"keywords": bd.get("keywords",[]), "filtered_text": bd.get("filtered_text","")}
# raw 백필: 종목코드가 확정된 뉴스는 종목명 인식 실패해도 보존 (관련성 과필터 완화)
kc = item.get("stock_code")
if kc and not any(st.get("code") == kc for st in data["stocks"]):
data["stocks"] = [{"name": await _corp_name(kc), "code": kc}] + data["stocks"]
# 1.5. 종목 매칭 + 시장 키워드 둘 다 없으면 스킵 (LLM 비용 절감)
if not data["stocks"]:
full_text = (item.get("title","") or "") + " " + (item.get("content","") or "")
if not _has_market_relevance(full_text):
return "off_topic"
# 2. Ollama 임베딩
er = await client.post(f"{OLLAMA_URL}/api/embeddings",
json={"model":"bge-m3","prompt": data["filtered_text"] or data["title"]}, timeout=60)
emb = er.json().get("embedding")
if not emb: return "no_embed"
# 3. Qdrant 유사도
try:
sr = await client.post(f"{QDRANT_URL}/collections/news_vectors/points/search",
json={"vector":emb,"limit":6,"score_threshold":0.80,"with_payload":True}, timeout=15)
hits = sr.json().get("result",[])
if any(h["score"]>=0.92 for h in hits): return "sim_dup" # ≥0.99 근접중복도 차단
except: hits = []
# 3.5. RAG 컨텍스트: 유사 과거뉴스 + 종목 재무·추세·점수 (버핏 판단 근거 주입)
ctx = []
rel = [h for h in hits if 0.80 <= h.get("score", 0) < 0.92][:4]
if rel:
ctx.append("· 유사 과거뉴스:")
for h in rel:
p = h.get("payload", {})
ctx.append(f" - {(p.get('title') or '')[:55]}"
f"{p.get('sentiment','?')}/강도{p.get('intensity','?')}")
focus = data["stocks"][0] if data["stocks"] else None
if focus:
g = lambda v: "-" if v is None else v
try:
fin = await pg_pool.fetchrow("""
SELECT ROUND(roe::numeric,1) roe, ROUND(operating_margin::numeric,1) om,
ROUND(debt_ratio::numeric,1) dr, ROUND(fcf_ratio::numeric,2) fcf,
ROUND(revenue_growth::numeric,1) rg
FROM dart_financials WHERE stock_code=$1 AND reprt_code='11011'
ORDER BY bsns_year DESC LIMIT 1""", focus["code"])
if fin:
ctx.append(f"· {focus['name']} 재무: ROE {g(fin['roe'])}% "
f"영업이익률 {g(fin['om'])}% 부채비율 {g(fin['dr'])}% "
f"FCF {g(fin['fcf'])} 매출성장 {g(fin['rg'])}%")
his = await pg_pool.fetchrow("""
SELECT COUNT(*) FILTER (WHERE sentiment='호재') pos,
COUNT(*) FILTER (WHERE sentiment='악재') neg,
ROUND(AVG(intensity)::numeric,1) ai,
MODE() WITHIN GROUP (ORDER BY catalyst) cat
FROM news_analysis
WHERE primary_stock=$1 AND analyzed_at >= NOW()-INTERVAL '14 days'
""", focus["code"])
if his and (his["pos"] or his["neg"]):
ctx.append(f"· 최근14일 {focus['name']}: 호재 {his['pos']} / "
f"악재 {his['neg']}, 평균강도 {g(his['ai'])}, "
f"주catalyst {his['cat'] or '-'}")
recent = await pg_pool.fetch("""
SELECT sentiment, intensity, catalyst, left(reason,50) reason
FROM news_analysis
WHERE primary_stock=$1 AND reason != '파싱실패'
AND analyzed_at >= NOW()-INTERVAL '30 days'
ORDER BY analyzed_at DESC LIMIT 2""", focus["code"])
if recent:
ctx.append(f"· {focus['name']} 최근 분석:")
for r in recent:
ctx.append(f" - {r['sentiment']}/강도{r['intensity']} "
f"[{r['catalyst'] or '-'}] {r['reason']}")
sc = await pg_pool.fetchrow("""
SELECT ROUND(total_score::numeric,1) total_score, recommendation
FROM stock_scores
WHERE stock_code=$1 ORDER BY score_date DESC LIMIT 1""",
focus["code"])
if sc:
ctx.append(f"· 현재 퀀트: 종합점수 {g(sc['total_score'])} "
f"({sc['recommendation'] or '-'})")
except Exception as e:
logger.debug("rag.ctx_err", code=focus["code"], error=str(e))
context_block = "\n".join(ctx)
# 4. Ollama EXAONE 분석 (버핏 관점 강화 프롬프트)
stocks_str = ", ".join([f'{s["name"]}({s["code"]})' for s in data["stocks"][:5]])
source = item.get("source", "")
content_preview = (data.get("filtered_text") or item.get("content") or "")[:400]
system_prompt = (
"당신은 워렌 버핏 스타일의 한국 주식 가치투자 애널리스트입니다.\n"
"뉴스 한 건을 읽고 기업 본질가치 관점에서 영향을 평가합니다.\n"
"판단 우선순위: 기업 본질가치(실적·FCF·경쟁우위) > 재무 리스크 > 단기 수급·테마\n\n"
"[sentiment] 호재 / 악재 / 중립 — 본질가치 또는 주가에 미치는 방향\n\n"
"[intensity] 1~5 정수\n"
" 5=상장폐지·대규모 횡령 등 기업 존폐\n"
" 4=연간실적 20% 이상 영향(대형 수주·어닝쇼크 등)\n"
" 3=분기실적에 유의미한 영향\n"
" 2=영업환경·업황 변화(직접 실적 수치 영향은 제한적)\n"
" 1=단순 정보성·정기공시·소폭 코멘트\n\n"
"[catalyst] 반드시 아래 6개 중 정확히 하나만 출력 (다른 단어·복합어·영문·한자 절대 금지):\n"
' "실적" = 영업이익·매출·순이익 증감, 흑자/적자 전환, 어닝 서프라이즈/쇼크, 실적 가이던스\n'
' "수주" = 신규 수주·공급계약·납품, M&A, 신사업·증설·신제품 등 성장 동인\n'
' "배당" = 배당 확대/축소, 자사주 매입·소각, 주주환원 정책\n'
' "리스크" = 횡령·배임·소송·감사의견거절·부채급증·유상증자(희석)·경영진 돌발사임·규제/제재·상폐위험\n'
' "모멘텀" = 본질가치 변화 없는 가격 동인(외국인/기관 수급, 목표주가 조정, 테마·정책 기대)\n'
' "기타" = 위 어디에도 명확히 속하지 않는 정보성 뉴스\n'
" 호재/악재면 catalyst를 '기타'로 두지 말고 실적·수주·배당·리스크·모멘텀 중 가장 가까운 것을 고르세요.\n\n"
"[참고] 블록은 보조 자료일 뿐 — 본 기사 내용으로 판단하고 과거 감성 흐름에 휩쓸리지 마세요.\n"
"동일 내용이 [참고]의 유사 과거뉴스에 반복되면 신규성이 낮으니 intensity를 보수적으로.\n\n"
"반드시 스키마에 맞는 유효한 JSON 객체 하나만 출력. 마크다운·설명문 금지."
)
user_prompt = (
f"[출처] {source}\n"
f"[제목] {data['title'][:200]}\n"
f"[내용] {content_preview}\n"
f"[관련종목] {stocks_str or '없음'}\n"
+ (f"\n[참고]\n{context_block}\n" if context_block else "")
+ "\ninvestment_action 규칙: 호재+intensity≥3→매수관심, 악재+intensity≥3→매도관심, 그 외→관망\n"
"primary_stock은 6자리 숫자코드(시장 전체 뉴스면 빈 문자열), "
"affected_stocks는 영향받는 다른 종목코드 배열, reason은 한 문장 핵심 근거.\n\n"
"JSON 스키마:\n"
'{"sentiment":"호재|악재|중립","intensity":1~5,"primary_stock":"005930",'
'"affected_stocks":["000660"],"reason":"핵심 근거 한 문장",'
'"investment_action":"매수관심|매도관심|관망",'
'"catalyst":"실적|수주|배당|리스크|모멘텀|기타"}\n'
"예시: "
'{"sentiment":"호재","intensity":4,"primary_stock":"005930",'
'"affected_stocks":["000660"],"reason":"HBM 대형 수주 확정으로 연간 반도체 실적 큰 폭 개선 기대",'
'"investment_action":"매수관심","catalyst":"수주"}'
)
# EXAONE 호출 + JSON 파싱 (실패 시 1회 재시도 → ~15% 유실 감소)
a = None
raw_prev = ""
for attempt in range(2):
msgs = [{"role":"system","content": system_prompt},
{"role":"user","content": user_prompt}]
if attempt == 1:
msgs += [{"role":"assistant","content": raw_prev[:600]},
{"role":"user","content":
"위 응답이 유효한 JSON이 아닙니다. 설명·마크다운 없이 "
"스키마에 맞는 JSON 객체 하나만 출력하세요."}]
try:
vr = await client.post(f"{OLLAMA_URL}/v1/chat/completions", json={
"model":"exaone3.5:7.8b","messages": msgs,
"max_tokens":400,"temperature":0.0 if attempt else 0.05},
timeout=120)
raw_prev = vr.json()["choices"][0]["message"]["content"]
c = raw_prev.replace("```json","").replace("```","").strip()
if not c.startswith("{"): # 앞뒤 설명 제거
s, e = c.find("{"), c.rfind("}")
if s != -1 and e > s: c = c[s:e+1]
a = json.loads(c)
break
except Exception:
continue
if a is None:
a = {"sentiment":"중립","intensity":0,"primary_stock":"","affected_stocks":[],"reason":"파싱실패","investment_action":"관망"}
# catalyst 6개 enum 강제 (score-engine CATALYST_WEIGHTS 정합 — 일탈값은 '기타')
if a.get("catalyst") not in ("실적","수주","배당","리스크","모멘텀","기타"):
a["catalyst"] = "기타"
# 5. Qdrant 저장
try:
await client.put(f"{QDRANT_URL}/collections/news_vectors/points", json={
"points":[{"id":random.randint(1,999999999),"vector":emb,
"payload":{"title":data["title"],"hash":data["hash"],
"sentiment":a.get("sentiment",""),"intensity":a.get("intensity",0),
"primary_stock":a.get("primary_stock","")}}]}, timeout=15)
except: pass
# 6. PostgreSQL 저장 (파라미터 바인딩 — 인젝션·이스케이프 유실 방지)
pub = data.get("published_at") or ""
try:
pub_dt = datetime.fromisoformat(pub.replace("Z", "+00:00")) if pub else datetime.now()
except Exception:
pub_dt = datetime.now()
try:
iv = int(a.get("intensity", 0) or 0)
except Exception:
iv = 0
s = lambda v: str(v if v is not None else "")
await pg_pool.execute("""
INSERT INTO news_analysis (title,url,source,published_at,hash,sentiment,intensity,
primary_stock,affected_stocks,reason,investment_action,keywords,stock_names,stock_codes,
catalyst,similar_count,analyzed_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9::jsonb,$10,$11,$12::jsonb,$13::jsonb,$14::jsonb,$15,0,NOW())
ON CONFLICT (hash) DO NOTHING
""",
s(data.get("title"))[:500], s(data.get("url"))[:500], s(data.get("source"))[:100],
pub_dt, data["hash"], s(a.get("sentiment", "중립"))[:10], iv,
s(a.get("primary_stock"))[:10], json.dumps(a.get("affected_stocks", []), ensure_ascii=False),
s(a.get("reason"))[:500], s(a.get("investment_action", "관망"))[:10],
json.dumps(data.get("keywords", [])[:20], ensure_ascii=False),
json.dumps([s_["name"] for s_ in data.get("stocks", [])], ensure_ascii=False),
json.dumps([s_["code"] for s_ in data.get("stocks", [])], ensure_ascii=False),
s(a.get("catalyst", "기타"))[:20])
return "ok"
except Exception as e:
logger.warning("pipeline.err", title=item.get("title","")[:50], error=str(e))
return "error"
# ── 수집 작업 ──────────────────────────────────────────────
async def job_rss():
"""RSS 28개 소스 수집 (5분마다, 2단계 중복제거)"""
if stats.running: return
stats.running = True
try:
async with httpx.AsyncClient() as c:
news = await crawl_rss_sources(c)
ok = 0
for item in news:
if await is_dup(item["title"], item.get("url", "")):
stats.duplicates += 1
continue
stats.collected += 1
r = await pipeline(item, c)
if r == "ok":
ok += 1
stats.processed += 1
elif r == "noise":
stats.noise += 1
elif r == "off_topic":
stats.off_topic += 1
stats.last_run = datetime.now().isoformat()
logger.info("job.rss", sources=len(RSS_SOURCES), total=len(news), processed=ok,
duplicates=stats.duplicates)
except Exception as e:
stats.errors += 1
logger.error("job.rss.err", error=str(e))
finally:
stats.running = False
NAVER_FINANCE_URLS = [
("https://finance.naver.com/news/mainnews.naver", "네이버금융"),
("https://finance.naver.com/news/news_list.nhn?mode=RANK&type=now", "네이버금융"),
]
async def crawl_naver_market(client) -> list:
news = []
for url, source in NAVER_FINANCE_URLS:
try:
r = await client.get(url, timeout=15, headers=HEADERS)
if r.status_code != 200:
continue
soup = BeautifulSoup(r.text, "html.parser")
anchors = (
soup.select("ul.newsList li a") or
soup.select(".headline_list li a") or
soup.select("a[href*='news_read']")
)
for a in anchors[:30]:
title = a.get_text(strip=True)
href = a.get("href", "")
if not href or not title or len(title) < 5:
continue
if href.startswith("/"):
href = f"https://finance.naver.com{href}"
if not is_korean(title):
continue
news.append({"title": title, "url": href, "source": source,
"content": "", "published_at": datetime.now().isoformat()})
except Exception as e:
logger.debug("naver.market.err", url=url, error=str(e))
logger.info("naver.market.crawled", items=len(news))
return news
async def crawl_naver_stock_news(client, code: str, max_pages: int = 20, sleep_s: float = 0.3) -> list:
"""네이버 모바일 종목 뉴스 API (JSON) — 백워드 페이징"""
items = []
for page in range(1, max_pages + 1):
url = f"https://m.stock.naver.com/api/news/stock/{code}?pageSize=20&page={page}"
try:
r = await client.get(url, headers=HEADERS, timeout=15)
if r.status_code != 200:
break
data = r.json()
if not isinstance(data, list) or not data:
break
page_items = data[0].get("items", []) if isinstance(data[0], dict) else []
if not page_items:
break
for it in page_items:
title = (it.get("title") or "").strip()
article_id = it.get("articleId") or ""
office_id = it.get("officeId") or ""
if not title or not article_id or not is_korean(title):
continue
url_news = f"https://n.news.naver.com/article/{office_id}/{article_id}"
dt = it.get("datetime") or ""
pub = (f"{dt[:4]}-{dt[4:6]}-{dt[6:8]} {dt[8:10]}:{dt[10:12]}"
if len(dt) >= 12 else "")
items.append({
"title": title, "url": url_news,
"source": it.get("officeName") or "네이버",
"published_at": pub,
"content": (it.get("body") or "")[:500],
})
await asyncio.sleep(sleep_s)
except Exception as e:
logger.debug("naver_mobile.err", code=code, page=page, error=str(e))
break
return items
async def save_raw_news(items: list, stock_code: str) -> int:
"""raw 뉴스 일괄 저장 (분석 스킵, 빠른 누적용)"""
if not items: return 0
saved = 0
async with pg_pool.acquire() as conn:
for it in items:
try:
url = it.get("url", "")
title = it.get("title", "")
if not url or not title: continue
url_hash = hashlib.sha256(url.encode()).hexdigest()
res = await conn.execute("""
INSERT INTO news_raw (stock_code, title, url, url_hash, source, published_at_text)
VALUES ($1,$2,$3,$4,$5,$6) ON CONFLICT (url_hash) DO NOTHING
""", stock_code, title, url, url_hash,
it.get("source", ""), it.get("published_at", ""))
if "INSERT 0 1" in res:
saved += 1
except Exception as e:
logger.debug("raw.save_err", error=str(e))
return saved
async def job_historical_raw(count: int = 0, max_pages: int = 100):
"""전체 활성종목 백워드 크롤링 → news_raw에 raw 저장 (LLM 분석 X, 빠름)
count=0 활성 종목 전체"""
if stats.running:
logger.warning("historical_raw.skip_running"); return
stats.running = True
try:
async with pg_pool.acquire() as conn:
if count > 0:
rows = await conn.fetch("""
SELECT stock_code FROM dart_corps WHERE is_active=true
ORDER BY stock_code LIMIT $1
""", count)
else:
rows = await conn.fetch(
"SELECT stock_code FROM dart_corps WHERE is_active=true ORDER BY stock_code")
codes = [r["stock_code"] for r in rows]
logger.info("historical_raw.start", n_stocks=len(codes), max_pages=max_pages)
total_saved = 0
async with httpx.AsyncClient() as c:
for i, code in enumerate(codes):
try:
items = await crawl_naver_stock_news(c, code, max_pages=max_pages)
saved = await save_raw_news(items, code)
total_saved += saved
if (i + 1) % 100 == 0:
logger.info("historical_raw.progress",
done=i+1, of=len(codes), total_saved=total_saved)
await asyncio.sleep(0.4)
except Exception as e:
stats.errors += 1
logger.warning("historical_raw.stock_err", code=code, error=str(e))
stats.last_run = datetime.now().isoformat()
logger.info("historical_raw.done", n_stocks=len(codes), total_saved=total_saved)
finally:
stats.running = False
async def job_process_raw(batch_size: int = 200):
"""news_raw의 미처리 행을 batch_size만큼 EXAONE 분석 → news_analysis로 이동"""
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT id, stock_code, title, url, source, published_at_text
FROM news_raw WHERE processed=false
ORDER BY id DESC LIMIT $1
""", batch_size)
if not rows:
logger.info("process_raw.empty"); return 0
ok = 0
async with httpx.AsyncClient() as c:
for r in rows:
item = {
"title": r["title"], "url": r["url"],
"source": r["source"] or "네이버금융",
"published_at": r["published_at_text"] or "",
"content": "",
"stock_code": r["stock_code"],
}
try:
if await is_dup(item["title"], item["url"]):
stats.duplicates += 1
else:
stats.collected += 1
res = await pipeline(item, c)
if res == "ok": ok += 1; stats.processed += 1
elif res == "noise": stats.noise += 1
elif res == "off_topic": stats.off_topic += 1
async with pg_pool.acquire() as conn:
await conn.execute(
"UPDATE news_raw SET processed=true, processed_at=NOW() WHERE id=$1",
r["id"])
except Exception as e:
stats.errors += 1
logger.warning("process_raw.err", id=r["id"], error=str(e))
logger.info("process_raw.batch_done", batch=len(rows), processed_ok=ok)
return ok
async def job_historical(count: int = 500, max_pages: int = 20):
"""과거 뉴스 백필: stock_scores 점수 매겨진 활성 종목 우선 + 종목당 max_pages 페이지"""
if stats.running:
logger.warning("historical.skip_running")
return
stats.running = True
try:
async with pg_pool.acquire() as conn:
# 우선순위: 최근 점수 매겨진 종목 → 그 외 활성 종목 alphabetical
rows = await conn.fetch("""
WITH scored AS (
SELECT DISTINCT stock_code, MAX(score_date) AS last_d
FROM stock_scores GROUP BY stock_code
), priority AS (
SELECT d.stock_code,
CASE WHEN s.stock_code IS NOT NULL THEN 0 ELSE 1 END AS p,
COALESCE(s.last_d, '1900-01-01'::date) AS d
FROM dart_corps d LEFT JOIN scored s ON s.stock_code=d.stock_code
WHERE d.is_active=true
)
SELECT stock_code FROM priority
ORDER BY p ASC, d DESC, stock_code ASC LIMIT $1
""", count)
codes = [r["stock_code"] for r in rows]
logger.info("historical.start", n_stocks=len(codes), max_pages=max_pages)
ok = 0; total_items = 0
async with httpx.AsyncClient() as c:
for i, code in enumerate(codes):
try:
items = await crawl_naver_stock_news(c, code, max_pages=max_pages)
total_items += len(items)
for item in items:
if await is_dup(item["title"], item.get("url", "")):
stats.duplicates += 1
continue
stats.collected += 1
item["stock_code"] = code
r = await pipeline(item, c)
if r == "ok":
ok += 1; stats.processed += 1
elif r == "noise": stats.noise += 1
elif r == "off_topic": stats.off_topic += 1
if (i + 1) % 50 == 0:
logger.info("historical.progress", done=i+1, of=len(codes),
items=total_items, processed=ok)
await asyncio.sleep(0.5) # 종목 간격 (차단 회피)
except Exception as e:
stats.errors += 1
logger.warning("historical.stock_err", code=code, error=str(e))
stats.last_run = datetime.now().isoformat()
logger.info("historical.done", n_stocks=len(codes), items=total_items, processed=ok)
finally:
stats.running = False
async def job_market():
if stats.running:
return
stats.running = True
try:
async with httpx.AsyncClient() as c:
news = await crawl_naver_market(c)
ok = 0
for item in news:
if await is_dup(item["title"], item.get("url", "")):
stats.duplicates += 1
continue
stats.collected += 1
r = await pipeline(item, c)
if r == "ok":
ok += 1
stats.processed += 1
elif r == "noise":
stats.noise += 1
elif r == "off_topic":
stats.off_topic += 1
stats.last_run = datetime.now().isoformat()
logger.info("job.market", total=len(news), processed=ok, duplicates=stats.duplicates)
except Exception as e:
stats.errors += 1
logger.error("job.market.err", error=str(e))
finally:
stats.running = False
# ── FastAPI ────────────────────────────────────────────────
app = FastAPI(title="뉴스 수집기 (멀티소스)")
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
async def init_news_tables():
"""news_raw 테이블 (수집/분석 분리용 임시 저장소)"""
async with pg_pool.acquire() as conn:
await conn.execute("""
CREATE TABLE IF NOT EXISTS news_raw (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10),
title TEXT NOT NULL,
url TEXT NOT NULL,
url_hash VARCHAR(64) UNIQUE,
source VARCHAR(100),
published_at_text VARCHAR(50),
collected_at TIMESTAMP DEFAULT NOW(),
processed BOOLEAN DEFAULT FALSE,
processed_at TIMESTAMP
)
""")
await conn.execute(
"CREATE INDEX IF NOT EXISTS idx_news_raw_unprocessed ON news_raw(processed) WHERE processed=false")
await conn.execute(
"CREATE INDEX IF NOT EXISTS idx_news_raw_stock ON news_raw(stock_code)")
@app.on_event("startup")
async def startup():
global pg_pool, redis_cl
pg_pool = await asyncpg.create_pool(host=PG_HOST,port=PG_PORT,database=PG_DB,user=PG_USER,password=PG_PASS,min_size=2,max_size=5)
redis_cl = aioredis.Redis(host=REDIS_HOST,port=6379,password=REDIS_PASSWORD,db=4,decode_responses=True)
await init_news_tables()
# 평일 RSS: 8-18시 5분마다
scheduler.add_job(job_rss,"cron",day_of_week="mon-fri",hour="8-18",minute="*/5",id="rss_weekday",replace_existing=True)
# 주말 RSS: 8-22시 15분마다 (누적 학습용)
scheduler.add_job(job_rss,"cron",day_of_week="sat,sun",hour="8-22",minute="*/15",id="rss_weekend",replace_existing=True)
# 평일 마켓 (네이버 금융): 9-17시 10분마다
scheduler.add_job(job_market,"cron",day_of_week="mon-fri",hour="9-17",minute="*/10",id="market",replace_existing=True)
# raw 뉴스 분석: 24시간 30분마다 batch 200건 처리 (백로그 소화용)
scheduler.add_job(job_process_raw,"cron",hour="*",minute="*/30",
id="process_raw",replace_existing=True,kwargs={"batch_size":200})
# 매주 일요일 새벽 2시 raw 백필 (전체 활성종목)
scheduler.add_job(job_historical_raw,"cron",day_of_week="sun",hour="2",minute="0",
id="historical_raw_weekly",replace_existing=True,
kwargs={"count":0,"max_pages":50})
scheduler.start()
logger.info("news-collector.started", sources=len(RSS_SOURCES))
@app.on_event("shutdown")
async def shutdown():
scheduler.shutdown()
if pg_pool: await pg_pool.close()
if redis_cl: await redis_cl.aclose()
@app.get("/health")
async def health():
return JSONResponse(content={"status":"ok","collected":stats.collected,"processed":stats.processed,
"duplicates":stats.duplicates,"noise":stats.noise,"off_topic":stats.off_topic,
"errors":stats.errors,"last_run":stats.last_run,"running":stats.running})
@app.post("/collect/rss")
async def m_rss():
asyncio.create_task(job_rss()); return {"status":"started","type":"rss","sources":len(RSS_SOURCES)}
@app.post("/collect/market")
async def m_market():
asyncio.create_task(job_market()); return {"status":"started","type":"market"}
@app.post("/collect/historical")
async def m_historical(count: int = 500, max_pages: int = 20):
asyncio.create_task(job_historical(count, max_pages))
return {"status":"started","type":"historical","count":count,"max_pages":max_pages}
@app.post("/collect/historical-raw")
async def m_historical_raw(count: int = 0, max_pages: int = 100):
"""전체 활성종목 raw 백필 (LLM 분석 X). count=0 → 전체"""
asyncio.create_task(job_historical_raw(count, max_pages))
return {"status":"started","type":"historical_raw","count":count or "전체","max_pages":max_pages}
@app.post("/process/raw")
async def m_process_raw(batch_size: int = 200):
"""news_raw 미처리분 batch 분석 → news_analysis로 이동"""
n = await job_process_raw(batch_size)
return {"status":"done","processed":n}
@app.get("/raw/stats")
async def raw_stats():
"""raw 백필 진행 상태"""
async with pg_pool.acquire() as conn:
s = await conn.fetchrow("""
SELECT
COUNT(*) AS total,
COUNT(*) FILTER (WHERE processed=false) AS unprocessed,
COUNT(*) FILTER (WHERE processed=true) AS processed,
COUNT(DISTINCT stock_code) AS unique_stocks,
MIN(collected_at) AS first_collected,
MAX(collected_at) AS last_collected
FROM news_raw
""")
return dict(s) if s else {}
@app.get("/sources")
async def list_sources():
return [{"name": s, "url": u} for s, u in RSS_SOURCES]
+10
View File
@@ -0,0 +1,10 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
httpx==0.27.0
redis==5.0.4
asyncpg==0.29.0
beautifulsoup4==4.12.3
lxml==5.2.2
apscheduler==3.10.4
orjson==3.10.3
structlog==24.2.0
+19
View File
@@ -0,0 +1,19 @@
service:
http_port: 6333
grpc_port: 6334
max_request_size_mb: 32
storage:
performance:
max_search_threads: 4
max_optimization_threads: 2
optimizers:
deleted_threshold: 0.2
vacuum_min_vector_number: 1000
default_segment_number: 4
indexing_threshold_kb: 20000
flush_interval_sec: 5
max_optimization_threads: 2
log_level: INFO
+10
View File
@@ -0,0 +1,10 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
httpx==0.27.0
redis==5.0.4
asyncpg==0.29.0
beautifulsoup4==4.12.3
lxml==5.2.2
apscheduler==3.10.4
orjson==3.10.3
structlog==24.2.0
+8
View File
@@ -0,0 +1,8 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8686
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8686", "--workers", "1", "--log-level", "info"]
+3155
View File
File diff suppressed because it is too large Load Diff
+13
View File
@@ -0,0 +1,13 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
httpx==0.27.0
asyncpg==0.29.0
redis==5.0.4
apscheduler==3.10.4
orjson==3.10.3
structlog==24.2.0
scikit-learn==1.5.0
numpy==1.26.4
xgboost==2.0.3
arch==7.0.0
scipy==1.13.1
+146
View File
@@ -0,0 +1,146 @@
#!/bin/bash
# PostgreSQL 로컬 설치 + 데이터 경로를 시놀로지 NAS(/mnt/nas)에 설정
# 실행: sudo bash setup-postgres-local.sh
set -e
NAS_HOST="192.168.0.36"
NAS_NFS_PATH="/volume1/trading"
NFS_MOUNT_LOCAL="/mnt/nas"
PG_DATA_DIR="${NFS_MOUNT_LOCAL}/postgresql/data"
PG_VERSION="16"
PG_USER="kyu"
PG_PASSWORD="7895123"
PG_DB="trading_ai"
echo "=============================="
echo "1. NFS 클라이언트 설치 확인"
echo "=============================="
apt-get install -y nfs-common 2>/dev/null || true
echo ""
echo "=============================="
echo "2. NAS NFS 마운트"
echo "=============================="
mkdir -p "${NFS_MOUNT_LOCAL}"
# 이미 마운트 되어있으면 스킵
if mountpoint -q "${NFS_MOUNT_LOCAL}"; then
echo "이미 마운트됨: ${NFS_MOUNT_LOCAL}"
else
mount -t nfs "${NAS_HOST}:${NAS_NFS_PATH}" "${NFS_MOUNT_LOCAL}" -o rw,hard,intr,timeo=30,retrans=3
echo "마운트 완료: ${NAS_HOST}:${NAS_NFS_PATH} -> ${NFS_MOUNT_LOCAL}"
fi
# /etc/fstab에 자동 마운트 등록 (없으면 추가)
FSTAB_ENTRY="${NAS_HOST}:${NAS_NFS_PATH} ${NFS_MOUNT_LOCAL} nfs rw,hard,intr,timeo=30,retrans=3,_netdev 0 0"
if ! grep -qF "${NFS_MOUNT_LOCAL}" /etc/fstab; then
echo "${FSTAB_ENTRY}" >> /etc/fstab
echo "fstab에 자동 마운트 추가됨"
fi
echo ""
echo "=============================="
echo "3. PostgreSQL 데이터 디렉토리 생성"
echo "=============================="
mkdir -p "${PG_DATA_DIR}"
chown postgres:postgres "${PG_DATA_DIR}"
chmod 700 "${PG_DATA_DIR}"
echo "데이터 경로: ${PG_DATA_DIR}"
echo ""
echo "=============================="
echo "4. PostgreSQL 서비스 설정"
echo "=============================="
# 기존 클러스터 중지 및 제거
pg_ctlcluster ${PG_VERSION} main stop 2>/dev/null || true
# 기존 기본 데이터 디렉토리를 NAS 경로로 교체
CONF_FILE="/etc/postgresql/${PG_VERSION}/main/postgresql.conf"
if [ -f "${CONF_FILE}" ]; then
# data_directory 변경
sed -i "s|^data_directory.*|data_directory = '${PG_DATA_DIR}'|" "${CONF_FILE}"
# 이미 변경된 경우가 아니면 추가
if ! grep -q "^data_directory" "${CONF_FILE}"; then
echo "data_directory = '${PG_DATA_DIR}'" >> "${CONF_FILE}"
fi
echo "postgresql.conf 업데이트 완료"
fi
echo ""
echo "=============================="
echo "5. 데이터 디렉토리 초기화 (최초 1회)"
echo "=============================="
if [ ! -f "${PG_DATA_DIR}/PG_VERSION" ]; then
sudo -u postgres /usr/lib/postgresql/${PG_VERSION}/bin/initdb \
-D "${PG_DATA_DIR}" \
--encoding=UTF8 \
--locale=ko_KR.UTF-8 \
--auth=md5 2>/dev/null || \
sudo -u postgres /usr/lib/postgresql/${PG_VERSION}/bin/initdb \
-D "${PG_DATA_DIR}" \
--encoding=UTF8 \
--locale=C.UTF-8 \
--auth=md5
echo "DB 초기화 완료"
else
echo "이미 초기화된 데이터 디렉토리 (스킵)"
fi
echo ""
echo "=============================="
echo "6. 외부 접속 허용 설정"
echo "=============================="
# postgresql.conf - 모든 인터페이스 리슨
sed -i "s|^#listen_addresses.*|listen_addresses = '*'|; s|^listen_addresses.*|listen_addresses = '*'|" "${PG_DATA_DIR}/postgresql.conf" 2>/dev/null || \
echo "listen_addresses = '*'" >> "${PG_DATA_DIR}/postgresql.conf"
# pg_hba.conf - Docker 네트워크(172.30.0.0/16) 접속 허용
PG_HBA="${PG_DATA_DIR}/pg_hba.conf"
if ! grep -q "172.30.0.0/16" "${PG_HBA}" 2>/dev/null; then
cat >> "${PG_HBA}" << 'HBA'
# Trading AI Docker 네트워크
host all all 172.30.0.0/16 md5
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
HBA
echo "pg_hba.conf 업데이트 완료"
fi
echo ""
echo "=============================="
echo "7. PostgreSQL 서비스 시작"
echo "=============================="
systemctl start postgresql
systemctl enable postgresql
sleep 3
systemctl is-active postgresql && echo "PostgreSQL 실행 중"
echo ""
echo "=============================="
echo "8. 유저 및 DB 생성"
echo "=============================="
sudo -u postgres psql -c "DO \$\$ BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '${PG_USER}') THEN
CREATE USER ${PG_USER} WITH SUPERUSER PASSWORD '${PG_PASSWORD}';
ELSE
ALTER USER ${PG_USER} WITH PASSWORD '${PG_PASSWORD}';
END IF;
END \$\$;"
sudo -u postgres psql -c "CREATE DATABASE ${PG_DB} OWNER ${PG_USER};" 2>/dev/null || \
echo "DB ${PG_DB} 이미 존재 (스킵)"
echo ""
echo "=============================="
echo "완료! PostgreSQL 로컬 실행 중"
echo " Host: localhost (또는 127.0.0.1)"
echo " Port: 5432"
echo " DB: ${PG_DB}"
echo " User: ${PG_USER}"
echo " Data: ${PG_DATA_DIR} (NAS)"
echo "=============================="
echo ""
echo "다음 단계: 덤프 임포트"
echo " PGPASSWORD=${PG_PASSWORD} psql -h 127.0.0.1 -U ${PG_USER} -d ${PG_DB} -f /home/kyu/trading/pg_backup/trading_ai_backup.sql"
+111
View File
@@ -0,0 +1,111 @@
#!/usr/bin/env bash
# ============================================================
# Trading AI - Setup Script
# ============================================================
set -euo pipefail
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
log() { echo -e "${GREEN}[✓]${NC} $1"; }
warn() { echo -e "${YELLOW}[!]${NC} $1"; }
err() { echo -e "${RED}[✗]${NC} $1" >&2; exit 1; }
[ -f .env ] && source .env || err ".env 파일이 없습니다."
echo "============================================================"
echo " Trading AI System - Setup"
echo "============================================================"
# ── 1. Docker / NVIDIA 확인 ────────────────────────────────
command -v docker >/dev/null 2>&1 || err "Docker가 없습니다."
if ! command -v nvidia-smi >/dev/null 2>&1; then
err "NVIDIA 드라이버가 없습니다."
fi
if ! docker info 2>/dev/null | grep -q "Runtimes.*nvidia"; then
warn "nvidia-container-toolkit 설치 중..."
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update -qq
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
log "nvidia-container-toolkit 설치 완료"
fi
log "GPU 정보:"
nvidia-smi --query-gpu=index,name,memory.total --format=csv,noheader | \
while IFS=, read -r idx name mem; do echo " GPU $idx: $name ($mem)"; done
# ── 2. 시스템 최적화 ────────────────────────────────────────
if ! grep -q "vm.overcommit_memory" /etc/sysctl.conf 2>/dev/null; then
printf "vm.overcommit_memory = 1\nvm.swappiness = 10\nnet.core.somaxconn = 65535\n" | \
sudo tee -a /etc/sysctl.conf
sudo sysctl -p >/dev/null 2>&1
log "커널 파라미터 최적화"
fi
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled >/dev/null
# ── 3. NFS 마운트 (스킵 - 나중에 수동 설정) ───────────────
warn "NFS 마운트 스킵 (Synology 설정 후 수동으로 진행하세요)"
sudo mkdir -p /mnt/nas/news /mnt/nas/models /mnt/nas/backups
# ── 4. 기존 네트워크 정리 ──────────────────────────────────
log "기존 컨테이너/네트워크 정리 중..."
docker compose down --remove-orphans 2>/dev/null || true
docker network prune -f >/dev/null 2>&1 || true
# ── 5. 순차 시작 ───────────────────────────────────────────
log "1단계: Redis, Qdrant, Bareun 시작..."
docker compose up -d --build redis qdrant bareun
log "헬스체크 대기 (20초)..."
sleep 20
log "2단계: bareunaapi 시작..."
docker compose up -d --build bareunaapi
log "3단계: Ollama 시작 (GPU 1 - RTX 3070)..."
docker compose up -d ollama
log "BGE-M3 모델 다운로드 중..."
sleep 30
docker exec trading-ollama ollama pull bge-m3 2>/dev/null || \
warn "BGE-M3 수동 다운로드 필요: docker exec trading-ollama ollama pull bge-m3"
log "4단계: vLLM 시작 (GPU 0 - RTX 3060, 모델 로딩 2~5분)..."
docker compose up -d vllm
log "5단계: n8n 시작..."
docker compose up -d n8n n8n-worker
# ── 6. Qdrant 컬렉션 초기화 ────────────────────────────────
log "Qdrant 컬렉션 초기화 중..."
sleep 10
HTTP=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:6333/collections/news_vectors)
if [ "${HTTP}" = "404" ]; then
curl -s -X PUT http://localhost:6333/collections/news_vectors \
-H "Content-Type: application/json" \
-d '{
"vectors": {"size": 1024, "distance": "Cosine"},
"optimizers_config": {"default_segment_number": 4},
"on_disk_payload": false
}' >/dev/null
log "Qdrant 컬렉션 'news_vectors' 생성"
else
warn "Qdrant 컬렉션 이미 존재"
fi
echo ""
echo "============================================================"
echo -e "${GREEN} 시작 완료!${NC}"
echo "============================================================"
echo " n8n : http://localhost:5678"
echo " Qdrant : http://localhost:6333/dashboard"
echo " vLLM : http://localhost:8000/docs"
echo " Ollama : http://localhost:11434"
echo " 바른API : http://localhost:5757/docs"
echo "============================================================"
warn "vLLM 모델 로딩 확인: docker compose logs -f vllm"
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env bash
[ -f .env ] && source .env
GREEN='\033[0;32m'; RED='\033[0;31m'; YELLOW='\033[1;33m'; BLUE='\033[0;34m'; NC='\033[0m'
chk_http() {
local name=$1 url=$2
code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 4 "$url" 2>/dev/null || echo "000")
[[ "$code" =~ ^(200|204)$ ]] \
&& echo -e " ${GREEN}${NC} $name" \
|| echo -e " ${RED}${NC} $name (HTTP $code)"
}
chk_container() {
local name=$1
status=$(docker inspect --format='{{.State.Status}}' "$name" 2>/dev/null || echo "없음")
[ "$status" = "running" ] \
&& echo -e " ${GREEN}${NC} $name" \
|| echo -e " ${RED}${NC} $name ($status)"
}
echo ""
echo -e "${BLUE}══ Trading AI Status ══════════════════════${NC} $(date '+%Y-%m-%d %H:%M:%S')"
echo -e "\n${BLUE}[컨테이너]${NC}"
for c in trading-redis trading-qdrant trading-bareun trading-bareunaapi \
trading-ollama trading-vllm trading-n8n trading-n8n-worker; do
chk_container "$c"
done
echo -e "\n${BLUE}[서비스]${NC}"
chk_http "Redis" "http://localhost:6379"
chk_http "Qdrant" "http://localhost:6333/healthz"
chk_http "Bareun" "http://localhost:9902/health"
chk_http "바른API" "http://localhost:5757/health"
chk_http "Ollama" "http://localhost:11434/api/tags"
chk_http "vLLM" "http://localhost:8000/health"
chk_http "n8n" "http://localhost:5678/healthz"
echo -e "\n${BLUE}[GPU]${NC}"
nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu \
--format=csv,noheader 2>/dev/null | \
while IFS=, read -r i n u mu mt t; do
echo " GPU$i $n | 사용률:$u | VRAM:$mu/$mt | 온도:$t"
done
echo -e "\n${BLUE}[NFS]${NC}"
mountpoint -q /mnt/nas 2>/dev/null \
&& echo -e " ${GREEN}${NC} /mnt/nas (마운트됨)" \
|| echo -e " ${RED}${NC} /mnt/nas (마운트 안됨)"
echo ""
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/env bash
# ============================================================
# Phase 1~4 전체 설치 스크립트
# ============================================================
set -euo pipefail
GREEN='\033[0;32m'; NC='\033[0m'
log() { echo -e "${GREEN}[✓]${NC} $1"; }
cd ~/trading
# ── 1. 디렉토리 생성 ───────────────────────────────────
log "디렉토리 생성..."
mkdir -p news-collector kis-api score-engine dashboard-api
# ── 2. .env 추가 ────────────────────────────────────────
log ".env 업데이트..."
grep -q "KIS_APP_KEY" .env 2>/dev/null || cat >> .env << 'EOF'
# Phase 2: 한국투자증권 (없으면 빈값)
KIS_APP_KEY=
KIS_APP_SECRET=
KIS_ACCOUNT_NO=
KIS_IS_PAPER=true
EOF
# ── 3. 빌드 및 시작 ─────────────────────────────────────
log "Phase 1: 뉴스 수집기 빌드..."
docker compose up -d --build news-collector
log "Phase 2: KIS API 빌드..."
docker compose up -d --build kis-api
log "Phase 3: 점수 엔진 빌드..."
docker compose up -d --build score-engine
log "Phase 4: 대시보드 API 빌드..."
docker compose up -d --build dashboard-api
# ── 4. 헬스체크 ──────────────────────────────────────────
log "서비스 시작 대기 (30초)..."
sleep 30
echo ""
echo "============================================================"
echo -e "${GREEN} Phase 1~4 설치 완료!${NC}"
echo "============================================================"
echo ""
echo " 서비스 목록:"
echo " ─────────────────────────────────────────"
echo " 뉴스 수집기 : http://192.168.0.60:8787"
echo " KIS 주가 API : http://192.168.0.60:8585"
echo " 점수 엔진 : http://192.168.0.60:8686"
echo " 대시보드 API : http://192.168.0.60:8989"
echo " DART 수집기 : http://192.168.0.60:8888"
echo " n8n : http://192.168.0.60:5678"
echo " 바른API : http://192.168.0.60:5757"
echo " vLLM : http://192.168.0.60:8000"
echo ""
echo " 사용법:"
echo " ─────────────────────────────────────────"
echo " # 뉴스 수동 수집"
echo " curl -s -X POST http://localhost:8787/collect/market"
echo " curl -s -X POST http://localhost:8787/collect/stocks"
echo ""
echo " # 점수 수동 산출"
echo " curl -s -X POST http://localhost:8686/score/calculate"
echo ""
echo " # 추천 종목 조회"
echo " curl -s http://localhost:8686/recommendations | python3 -m json.tool"
echo ""
echo " # 종목 랭킹"
echo " curl -s http://localhost:8686/ranking | python3 -m json.tool"
echo ""
echo " # 종목 상세 (삼성전자)"
echo " curl -s http://localhost:8686/stock/005930 | python3 -m json.tool"
echo ""
echo " # 대시보드 API"
echo " curl -s http://localhost:8989/api/summary | python3 -m json.tool"
echo " curl -s http://localhost:8989/api/ranking | python3 -m json.tool"
echo " curl -s http://localhost:8989/api/recommendations | python3 -m json.tool"
echo "============================================================"
docker ps -a --format "table {{.Names}}\t{{.Status}}"
+162
View File
@@ -0,0 +1,162 @@
"""
KRX 전체 종목 동적 로딩
- 서버 시작 1 로딩
- 매일 자정 자동 갱신
- 실패 하드코딩 폴백
"""
import asyncio
import json
import io
import time
from datetime import datetime
import httpx
import structlog
logger = structlog.get_logger()
# 폴백 종목 (네트워크 실패 시 사용)
FALLBACK_STOCKS = {
"코스피": "KOSPI", "코스닥": "KOSDAQ", "코스피200": "KOSPI200",
"삼성전자": "005930", "SK하이닉스": "000660", "LG에너지솔루션": "373220",
"현대차": "005380", "기아": "000270", "셀트리온": "068270",
"카카오": "035720", "네이버": "035420", "NAVER": "035420",
"삼성바이오로직스": "207940", "KB금융": "105560", "POSCO홀딩스": "005490",
"신한지주": "055550", "LG화학": "051910", "삼성SDI": "006400",
"현대모비스": "012330", "하나금융지주": "086790", "SK텔레콤": "017670",
"KT": "030200", "LG전자": "066570", "한화에어로스페이스": "012450",
"삼성물산": "028260", "HD현대중공업": "329180", "한국전력": "015760",
"HMM": "011200", "대한항공": "003490", "카카오뱅크": "323410",
"크래프톤": "259960", "하이브": "352820", "에코프로": "086520",
"알테오젠": "196170", "한미반도체": "042700",
}
# 약칭/별칭
ALIASES = {
"삼전": "005930", "하닉": "000660", "현차": "005380",
"카뱅": "323410", "삼바": "207940", "삼성바이오": "207940",
"한에솔": "012450", "한화에어": "012450", "LG엔솔": "373220",
"SK하닉": "000660", "포홀": "005490", "에프엔에프": "383220",
"현대자동차": "005380", "네이버": "035420",
}
async def fetch_krx_stocks() -> dict[str, str]:
"""KRX에서 전체 상장 종목 가져오기"""
stock_map = {
"코스피": "KOSPI", "코스닥": "KOSDAQ",
"코스피200": "KOSPI200", "KRX": "KRX",
}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Referer": "http://data.krx.co.kr/contents/MDC/MDI/mdiLoader/index.cmd",
}
async with httpx.AsyncClient(timeout=30, headers=headers) as client:
for mkt_id, mkt_name in [("STK", "KOSPI"), ("KSQ", "KOSDAQ")]:
try:
payload = {
"bld": "dbms/MDC/STAT/standard/MDCSTAT01901",
"locale": "ko_KR",
"mktId": mkt_id,
"share": "1",
"csvxls_is498": "false",
}
resp = await client.post(
"http://data.krx.co.kr/comm/bldAttend/getJsonData.cmd",
data=payload,
)
if resp.status_code == 200:
data = resp.json()
items = data.get("OutBlock_1", [])
for item in items:
name = item.get("ISU_ABBRV", "").strip()
code = item.get("ISU_SRT_CD", "").strip()
if name and code and len(code) == 6:
stock_map[name] = code
logger.info("krx.loaded", market=mkt_name, count=len(items))
else:
logger.warning("krx.http_error", market=mkt_name, status=resp.status_code)
except Exception as e:
logger.warning("krx.fetch_failed", market=mkt_name, error=str(e))
# KRX 실패 시 네이버 금융 폴백
if len(stock_map) < 100:
logger.info("krx.fallback_naver", reason="KRX returned too few stocks")
stock_map.update(await fetch_naver_stocks(client=None))
return stock_map
async def fetch_naver_stocks() -> dict[str, str]:
"""네이버 금융에서 종목 가져오기 (KRX 폴백)"""
stock_map = {}
headers = {"User-Agent": "Mozilla/5.0"}
import re
async with httpx.AsyncClient(timeout=15, headers=headers) as client:
for sosok in [0, 1]: # 0=KOSPI, 1=KOSDAQ
for page in range(1, 45):
try:
url = f"https://finance.naver.com/sise/sise_market_sum.naver?sosok={sosok}&page={page}"
resp = await client.get(url)
text = resp.content.decode("euc-kr", errors="ignore")
rows = re.findall(
r"main\.naver\?code=(\d{6})[^>]*>([^<]+)</a>", text
)
if not rows:
break
for code, name in rows:
name = name.strip()
if name and code:
stock_map[name] = code
except Exception:
break
logger.info("naver.loaded", count=len(stock_map))
return stock_map
async def load_all_stocks() -> dict[str, str]:
"""전체 종목 로딩 (KRX → 네이버 → 폴백)"""
try:
stock_map = await fetch_krx_stocks()
if len(stock_map) > 100:
stock_map.update(ALIASES)
logger.info("stocks.loaded", total=len(stock_map), source="KRX")
return stock_map
except Exception as e:
logger.warning("stocks.krx_failed", error=str(e))
try:
stock_map = await fetch_naver_stocks()
if len(stock_map) > 100:
stock_map.update({
"코스피": "KOSPI", "코스닥": "KOSDAQ",
"코스피200": "KOSPI200", "KRX": "KRX",
})
stock_map.update(ALIASES)
logger.info("stocks.loaded", total=len(stock_map), source="Naver")
return stock_map
except Exception as e:
logger.warning("stocks.naver_failed", error=str(e))
# 최종 폴백
fallback = {**FALLBACK_STOCKS, **ALIASES}
logger.warning("stocks.using_fallback", total=len(fallback))
return fallback
async def auto_refresh_stocks(state_ref, interval_hours: int = 24):
"""백그라운드에서 종목 목록 자동 갱신"""
while True:
await asyncio.sleep(interval_hours * 3600)
try:
new_map = await load_all_stocks()
if len(new_map) > 50:
state_ref.stock_map = new_map
logger.info("stocks.refreshed", total=len(new_map))
except Exception as e:
logger.warning("stocks.refresh_failed", error=str(e))
+6
View File
@@ -0,0 +1,6 @@
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8484", "--workers", "1"]
+965
View File
@@ -0,0 +1,965 @@
"""
기술적 분석 엔진 (Technical Analysis Engine)
- 네이버 금융 차트 API (1) / yfinance (2 백업) OHLCV 수집
- MA5/20/60/120, RSI(14), MACD(12,26,9), 볼린저밴드(20,2), 스토캐스틱(14,3)
- 기술적 점수 (-100~100) 산출
- 매수/매도 목표가 T1/T2/T3 + 손절가 자동 계산
- vLLM AI 문장형 판단 생성
- 보유 포지션 손익 + 맞춤 전략 분석
"""
import asyncio, json, os, re, math
from datetime import datetime
from typing import Optional, List, Dict
import asyncpg, httpx, redis.asyncio as aioredis, structlog
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from fastapi import FastAPI, Query, Body
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
structlog.configure(processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(),
])
logger = structlog.get_logger()
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
PG_HOST = os.getenv("POSTGRES_HOST", "postgres")
PG_PORT = int(os.getenv("POSTGRES_PORT", "5432"))
PG_DB = os.getenv("POSTGRES_DB", "trading_ai")
PG_USER = os.getenv("POSTGRES_USER", "kyu")
PG_PASS = os.getenv("POSTGRES_PASSWORD", "")
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://ollama:11434")
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
pg_pool: Optional[asyncpg.Pool] = None
redis_cl: Optional[aioredis.Redis] = None
scheduler = AsyncIOScheduler(timezone="Asia/Seoul")
class Stats:
analyzed = 0; errors = 0; last_run = ""
stats = Stats()
# ── 기술적 지표 계산 ──────────────────────────────────────
def _ema_series(values: List[float], period: int) -> List[float]:
if not values or len(values) < period:
return [values[-1]] * len(values) if values else []
k = 2.0 / (period + 1)
seed = sum(values[:period]) / period
out = [seed]
for v in values[period:]:
out.append(v * k + out[-1] * (1 - k))
return [out[0]] * (period - 1) + out
def _ma(closes: List[float], n: int) -> float:
if not closes: return 0.0
data = closes[-n:] if len(closes) >= n else closes
return sum(data) / len(data)
def _rsi(closes: List[float], period: int = 14) -> float:
if len(closes) < period + 1:
return 50.0
deltas = [closes[i] - closes[i-1] for i in range(1, len(closes))]
gains = [max(d, 0.0) for d in deltas]
losses = [max(-d, 0.0) for d in deltas]
ag = sum(gains[:period]) / period
al = sum(losses[:period]) / period
for i in range(period, len(gains)):
ag = (ag * (period - 1) + gains[i]) / period
al = (al * (period - 1) + losses[i]) / period
if al == 0:
return 100.0
return round(100 - 100 / (1 + ag / al), 2)
def _macd(closes: List[float]) -> tuple:
if len(closes) < 26:
return 0.0, 0.0, 0.0
e12 = _ema_series(closes, 12)
e26 = _ema_series(closes, 26)
macd_line = [a - b for a, b in zip(e12, e26)]
signal_line = _ema_series(macd_line[-9:] if len(macd_line) >= 9 else macd_line, 9)
macd = macd_line[-1]
signal = signal_line[-1]
return round(macd, 4), round(signal, 4), round(macd - signal, 4)
def _bollinger(closes: List[float], period: int = 20) -> tuple:
if len(closes) < period:
c = closes[-1] if closes else 0
return float(c), float(c), float(c), 0.5
recent = closes[-period:]
ma = sum(recent) / period
std = math.sqrt(sum((x - ma) ** 2 for x in recent) / period)
upper = ma + 2 * std
lower = ma - 2 * std
cur = closes[-1]
pct_b = (cur - lower) / (upper - lower) if upper != lower else 0.5
return round(upper), round(ma), round(lower), round(pct_b, 3)
def _stochastic(highs: List[float], lows: List[float], closes: List[float], period: int = 14) -> tuple:
if len(closes) < period:
return 50.0, 50.0
h = max(highs[-period:])
l = min(lows[-period:])
k = ((closes[-1] - l) / (h - l) * 100) if h != l else 50.0
ks = []
for i in range(3):
idx = -(3 - i)
hh = max(highs[idx - period + 1:idx + 1] if idx != -1 else highs[-period:])
ll = min(lows[idx - period + 1:idx + 1] if idx != -1 else lows[-period:])
ks.append(((closes[idx] - ll) / (hh - ll) * 100) if hh != ll else 50.0)
d = sum(ks) / len(ks)
return round(k, 2), round(d, 2)
def _vol_ratio(volumes: List[float], period: int = 20) -> float:
if len(volumes) < period + 1:
return 1.0
avg = sum(volumes[-period - 1:-1]) / period
return round(volumes[-1] / avg, 2) if avg > 0 else 1.0
def _atr(highs: List[float], lows: List[float], closes: List[float], period: int = 14) -> float:
"""Average True Range — 변동성 측정"""
if len(closes) < period + 1:
return 0.0
trs = []
for i in range(1, len(closes)):
tr = max(
highs[i] - lows[i],
abs(highs[i] - closes[i-1]),
abs(lows[i] - closes[i-1]),
)
trs.append(tr)
recent = trs[-period:]
return round(sum(recent) / period, 2) if recent else 0.0
def _obv(closes: List[float], volumes: List[float]) -> tuple:
"""On-Balance Volume — 거래량 누적, 가격 상승일 +volume, 하락일 -volume"""
if len(closes) < 2: return 0, 0
obv = 0.0
obvs = [0.0]
for i in range(1, len(closes)):
if closes[i] > closes[i-1]: obv += volumes[i]
elif closes[i] < closes[i-1]: obv -= volumes[i]
obvs.append(obv)
# 최근 OBV 추세 (20일 평균 대비)
recent = obvs[-20:] if len(obvs) >= 20 else obvs
avg = sum(recent) / len(recent) if recent else 0
obv_trend = "상승" if obv > avg * 1.05 else ("하락" if obv < avg * 0.95 else "중립")
return int(obv), obv_trend
def _vwap(highs: List[float], lows: List[float], closes: List[float], volumes: List[float],
period: int = 20) -> float:
"""Volume Weighted Average Price — 거래량 가중 평균가, 기관 매매 기준선"""
if len(closes) < period: return float(closes[-1]) if closes else 0
cum_vp = sum(((highs[i] + lows[i] + closes[i]) / 3) * volumes[i]
for i in range(-period, 0))
cum_v = sum(volumes[-period:])
return round(cum_vp / cum_v, 2) if cum_v > 0 else 0
def _ichimoku(highs: List[float], lows: List[float], closes: List[float]) -> dict:
"""일목균형표 (Ichimoku Kinko Hyo) — 5개 라인"""
if len(closes) < 52: return {}
# 전환선(Tenkan-sen): (9일 고가 + 9일 저가) / 2
tenkan = (max(highs[-9:]) + min(lows[-9:])) / 2
# 기준선(Kijun-sen): (26일 고가 + 26일 저가) / 2
kijun = (max(highs[-26:]) + min(lows[-26:])) / 2
# 선행스팬1(Senkou Span A): (전환+기준)/2, 26일 후
span_a = (tenkan + kijun) / 2
# 선행스팬2(Senkou Span B): (52일 고가 + 52일 저가)/2, 26일 후
span_b = (max(highs[-52:]) + min(lows[-52:])) / 2
# 후행스팬(Chikou Span): 종가, 26일 전
chikou = closes[-26] if len(closes) > 26 else closes[-1]
cur = closes[-1]
cloud_top = max(span_a, span_b)
cloud_bot = min(span_a, span_b)
pos = "구름위" if cur > cloud_top else ("구름아래" if cur < cloud_bot else "구름속")
return {
"tenkan": int(tenkan), "kijun": int(kijun),
"span_a": int(span_a), "span_b": int(span_b), "chikou": int(chikou),
"cloud_pos": pos,
}
def calc_indicators(ohlcv: List[dict]) -> dict:
if len(ohlcv) < 5:
return {}
closes = [float(d["close"]) for d in ohlcv]
highs = [float(d["high"]) for d in ohlcv]
lows = [float(d["low"]) for d in ohlcv]
volumes = [float(d["volume"]) for d in ohlcv]
bb_upper, bb_mid, bb_lower, pct_b = _bollinger(closes, 20)
stoch_k, stoch_d = _stochastic(highs, lows, closes, 14)
macd, macd_signal, macd_hist = _macd(closes)
atr14 = _atr(highs, lows, closes, 14)
obv_val, obv_trend = _obv(closes, volumes)
vwap_val = _vwap(highs, lows, closes, volumes, 20)
ichi = _ichimoku(highs, lows, closes)
return {
"price": int(closes[-1]),
"ma5": round(_ma(closes, 5)),
"ma20": round(_ma(closes, 20)),
"ma60": round(_ma(closes, 60)),
"ma120": round(_ma(closes, 120)),
"rsi": _rsi(closes, 14),
"macd": macd,
"macd_signal": macd_signal,
"macd_hist": macd_hist,
"bb_upper": bb_upper,
"bb_mid": bb_mid,
"bb_lower": bb_lower,
"pct_b": pct_b,
"stoch_k": stoch_k,
"stoch_d": stoch_d,
"vol_ratio": _vol_ratio(volumes, 20),
"atr14": atr14,
"high_52w": int(max(highs[-min(len(highs), 252):])),
"low_52w": int(min(lows[-min(len(lows), 252):])),
"obv": obv_val,
"obv_trend": obv_trend,
"vwap20": vwap_val,
"ichimoku": ichi,
}
def calc_tech_score(ind: dict) -> tuple:
"""기술적 점수 (-100~100)와 근거 신호 목록 반환"""
if not ind:
return 0.0, []
price = ind["price"]
score = 0.0
signals: List[str] = []
# ── 이동평균 (±40) ──────────────────────────
if ind["ma5"] > ind["ma20"]:
score += 10; signals.append("MA5>MA20 단기상승")
else:
score -= 10; signals.append("MA5<MA20 단기하락")
if ind["ma20"] > ind["ma60"]:
score += 8; signals.append("MA20>MA60 중기상승")
else:
score -= 8
if ind["ma60"] > ind["ma120"]:
score += 7
else:
score -= 7
if price > ind["ma20"]:
score += 8; signals.append("현재가 MA20 위")
elif price < ind["ma60"]:
score -= 8; signals.append("현재가 MA60 아래")
# 정배열/역배열
if ind["ma5"] > ind["ma20"] > ind["ma60"] > ind["ma120"]:
score += 7; signals.append("정배열")
elif ind["ma5"] < ind["ma20"] < ind["ma60"] < ind["ma120"]:
score -= 7; signals.append("역배열")
# ── RSI (±25) ───────────────────────────────
rsi = ind["rsi"]
if rsi <= 30:
score += 25; signals.append(f"RSI 과매도({rsi:.0f})")
elif rsi <= 40:
score += 15; signals.append(f"RSI 저점({rsi:.0f})")
elif rsi <= 60:
score += 5
elif rsi <= 70:
score -= 5
else:
score -= 20; signals.append(f"RSI 과매수({rsi:.0f})")
# ── MACD (±20) ──────────────────────────────
if ind["macd_hist"] > 0 and ind["macd"] > ind["macd_signal"]:
score += 20; signals.append("MACD 골든크로스")
elif ind["macd_hist"] > 0:
score += 8
elif ind["macd_hist"] < 0 and ind["macd"] < ind["macd_signal"]:
score -= 20; signals.append("MACD 데드크로스")
else:
score -= 5
# ── 볼린저밴드 (±15) ────────────────────────
pb = ind["pct_b"]
if pb < 0.1:
score += 15; signals.append("볼밴 하단(과매도)")
elif pb < 0.3:
score += 8
elif pb > 0.9:
score -= 15; signals.append("볼밴 상단(과매수)")
elif pb > 0.7:
score -= 5
# ── 스토캐스틱 (±10) ────────────────────────
sk, sd = ind["stoch_k"], ind["stoch_d"]
if sk < 20 and sk > sd:
score += 10; signals.append("스토캐스틱 바닥반등")
elif sk > 80 and sk < sd:
score -= 10; signals.append("스토캐스틱 고점하락")
# ── 거래량 보너스 (±5) ──────────────────────
if ind["vol_ratio"] > 1.5:
if score > 0:
score += 5; signals.append("거래량 급증(매수세)")
else:
score -= 5; signals.append("거래량 급증(매도세)")
return round(max(-100.0, min(100.0, score)), 1), signals
def calc_price_targets(price: int, ind: dict, sig: str) -> dict:
"""매수/매도 목표가(T1/T2/T3) + 손절가 계산 (10원 단위 반올림)"""
if not ind or price <= 0:
return {}
def r10(p): return int(round(p / 10) * 10)
h52 = ind.get("high_52w", price * 1.3)
l52 = ind.get("low_52w", price * 0.7)
bb_up = ind.get("bb_upper", price * 1.05)
bb_dn = ind.get("bb_lower", price * 0.95)
ma20 = ind.get("ma20", price)
ma60 = ind.get("ma60", price)
if sig == "매수":
# 진입: 현재가 기준 -2% (기술지표 확인 후 매수)
entry = r10(price * 0.98)
# T1: +7% (단기), T2: +14% (중기), T3: min(+22%, 52주고가 -3%)
t1 = r10(price * 1.07)
t2 = r10(price * 1.14)
t3 = r10(min(price * 1.22, h52 * 0.97))
t3 = t3 if t3 > t2 else r10(price * 1.22)
# 손절: max(-8%, MA60 -5%) — 최대 -10% 이내 제한
raw_stop = max(price * 0.92, ma60 * 0.95)
stop = r10(max(raw_stop, price * 0.90)) # 최소 -10%
er1 = round((t1 - price) / price * 100, 1)
sl_r = round(abs(stop - price) / price * 100, 1)
# M3: ATR 기반 trailing stop (현재가 기준 2 ATR 아래)
atr = ind.get("atr14", 0)
atr_trailing = r10(price - 2 * atr) if atr > 0 else stop
return {
"entry_price": entry,
"t1": t1, "t1_pct": er1, "t1_sell_pct": 50,
"t2": t2, "t2_pct": round((t2 - price) / price * 100, 1), "t2_sell_pct": 30,
"t3": t3, "t3_pct": round((t3 - price) / price * 100, 1), "t3_sell_pct": 20,
"stop_loss": stop, "stop_pct": -sl_r,
"atr14": atr,
"trailing_stop": atr_trailing,
"risk_reward": round(er1 / sl_r, 2) if sl_r > 0 else 0,
"exit_strategy": "T1 50% + T2 30% + T3 20% 분할매도, 손절 또는 trailing(ATR×2) 도달시 전량",
}
else: # 매도 / 관망(음수)
entry = r10(price * 1.02)
t1 = r10(price * 0.93)
t2 = r10(price * 0.86)
t3 = r10(max(price * 0.78, l52 * 1.03))
t3 = t3 if t3 < t2 else r10(price * 0.78)
raw_stop = min(price * 1.08, ma20 * 1.05)
stop = r10(min(raw_stop, price * 1.10))
er1 = round((price - t1) / price * 100, 1)
sl_r = round(abs(stop - price) / price * 100, 1)
return {
"entry_price": entry,
"t1": t1, "t1_pct": -er1,
"t2": t2, "t2_pct": -round((price - t2) / price * 100, 1),
"t3": t3, "t3_pct": -round((price - t3) / price * 100, 1),
"stop_loss": stop, "stop_pct": sl_r,
"risk_reward": round(er1 / sl_r, 2) if sl_r > 0 else 0,
}
# ── OHLCV 수집 (네이버 차트 → yfinance 백업) ─────────────
async def get_ohlcv_naver_chart(client: httpx.AsyncClient, code: str, count: int = 120) -> List[dict]:
"""네이버 차트 API (fchart)"""
try:
r = await client.get(
f"https://fchart.stock.naver.com/sise.nhn?symbol={code}&timeframe=day&count={count}&requestType=0",
headers=HEADERS, timeout=10)
items = re.findall(r'data="([^"]+)"', r.text)
data = []
for item in items:
p = item.split("|")
if len(p) >= 6 and all(x.strip() for x in p[:5]):
try:
data.append({
"date": p[0], "open": int(p[1]), "high": int(p[2]),
"low": int(p[3]), "close": int(p[4]),
"volume": int(p[5]) if p[5].strip() else 0,
})
except ValueError:
pass
return data
except Exception:
return []
async def get_ohlcv_naver_sise(client: httpx.AsyncClient, code: str, pages: int = 7) -> List[dict]:
"""네이버 일별시세 페이지 (차트 API 실패 시 2차)"""
data = []
try:
for page in range(1, pages + 1):
r = await client.get(
f"https://finance.naver.com/item/sise_day.naver?code={code}&page={page}",
headers=HEADERS, timeout=12)
r.encoding = "euc-kr"
# 날짜+종가+전일비+시가+고가+저가+거래량 패턴
rows = re.findall(
r'(\d{4}\.\d{2}\.\d{2})[^<]*</span>.*?'
r'<span[^>]*>([\d,]+)</span>.*?' # 종가
r'(?:.*?){3}'
r'<span[^>]*>([\d,]+)</span>.*?' # 시가
r'<span[^>]*>([\d,]+)</span>.*?' # 고가
r'<span[^>]*>([\d,]+)</span>.*?' # 저가
r'<span[^>]*>([\d,]+)</span>', # 거래량
r.text, re.DOTALL)
if not rows:
# 단순 종가만 추출 (더 넓은 패턴)
simple = re.findall(
r'class="tah p10 gray03">(\d{4}\.\d{2}\.\d{2})<.*?'
r'class="tah p11">([\d,]+)<',
r.text, re.DOTALL)
for date_str, close_str in simple:
close = int(close_str.replace(",", ""))
if close > 0:
data.append({"date": date_str.replace(".", ""),
"open": close, "high": close,
"low": close, "close": close, "volume": 0})
else:
for m in rows:
date_str = m[0].replace(".", "")
close = int(m[1].replace(",", ""))
open_ = int(m[2].replace(",", "")) if m[2] else close
high = int(m[3].replace(",", "")) if m[3] else close
low = int(m[4].replace(",", "")) if m[4] else close
vol = int(m[5].replace(",", "")) if m[5] else 0
if close > 0:
data.append({"date": date_str, "open": open_,
"high": high, "low": low,
"close": close, "volume": vol})
if len(data) >= 120:
break
await asyncio.sleep(0.15)
except Exception as e:
logger.warning("ohlcv.sise.err", code=code, error=str(e))
return data[:120]
async def get_ohlcv_yfinance(code: str) -> List[dict]:
"""yfinance 최종 백업"""
try:
import yfinance as yf
loop = asyncio.get_event_loop()
def _fetch():
t = yf.Ticker(f"{code}.KS")
return t.history(period="1y")
h = await loop.run_in_executor(None, _fetch)
if h.empty:
return []
return [{"date": idx.strftime("%Y%m%d"),
"open": int(row["Open"] or 0), "high": int(row["High"] or 0),
"low": int(row["Low"] or 0), "close": int(row["Close"] or 0),
"volume": int(row["Volume"] or 0)}
for idx, row in h.iterrows() if row["Close"] and row["High"]]
except Exception as e:
logger.warning("ohlcv.yfinance.err", code=code, error=str(e))
return []
async def get_ohlcv(client: httpx.AsyncClient, code: str, count: int = 120) -> List[dict]:
data = await get_ohlcv_naver_chart(client, code, count)
if len(data) < 20:
data = await get_ohlcv_naver_sise(client, code)
if len(data) < 20:
logger.info("ohlcv.fallback.yfinance", code=code)
await asyncio.sleep(0.5) # rate limit 방지
data = await get_ohlcv_yfinance(code)
return data
# ── vLLM AI 판단문 생성 ───────────────────────────────────
async def generate_ai_opinion(client: httpx.AsyncClient, code: str, name: str,
ind: dict, tech_score: float, signals: List[str],
targets: dict, news_score: float = 0) -> str:
"""vLLM으로 문장형 투자 판단 생성"""
sig = "매수" if tech_score >= 30 else ("매도" if tech_score <= -30 else "관망")
price = ind.get("price", 0)
rsi = ind.get("rsi", 50)
h52 = ind.get("high_52w", price)
l52 = ind.get("low_52w", price)
pos52 = int((price - l52) / (h52 - l52) * 100) if h52 != l52 else 50
prompt = f"""다음 주식 데이터를 바탕으로 투자자에게 명확한 매매 판단을 3~5문장으로 설명하세요.
한국어로, 구체적인 가격과 수치를 포함해서 작성하세요.
종목: {name}({code})
현재가: {price:,}
기술점수: {tech_score} / 신호: {sig}
이동평균: MA5={ind.get('ma5',0):,} MA20={ind.get('ma20',0):,} MA60={ind.get('ma60',0):,}
RSI: {rsi} / MACD히스토그램: {'양수(골든)' if ind.get('macd_hist',0)>0 else '음수(데드)'}
볼린저%B: {ind.get('pct_b',0.5)*100:.0f}%
52주위치: 하단에서 {pos52}%
뉴스감성점수: {news_score:.0f}
기술신호: {', '.join(signals[:4])}
{f"1차목표가: {targets.get('t1',0):,}원 / 손절가: {targets.get('stop_loss',0):,}" if targets else ""}
투자 판단 (3~5문장):"""
try:
r = await client.post(f"{OLLAMA_URL}/v1/chat/completions", json={
"model": "exaone3.5:7.8b",
"messages": [
{"role": "system", "content": "당신은 한국 주식 전문 애널리스트입니다. 기술적 분석 데이터를 바탕으로 명확하고 실용적인 투자 의견을 제시합니다."},
{"role": "user", "content": prompt}
],
"max_tokens": 300, "temperature": 0.2
}, timeout=60)
return r.json()["choices"][0]["message"]["content"].strip()
except Exception as e:
logger.warning("ai_opinion.err", code=code, error=str(e))
return ""
# ── 포지션 손익 분석 ──────────────────────────────────────
class PositionRequest(BaseModel):
code: str
name: str = ""
buy_price: int
qty: int
def analyze_position(price: int, buy_price: int, qty: int,
ind: dict, tech_score: float) -> dict:
"""보유 포지션 기반 맞춤 전략 계산"""
pnl = (price - buy_price) * qty
pnl_pct = (price - buy_price) / buy_price * 100
total_buy = buy_price * qty
h52 = ind.get("high_52w", price * 1.3)
l52 = ind.get("low_52w", price * 0.7)
ma20 = ind.get("ma20", price)
ma60 = ind.get("ma60", price)
bbu = ind.get("bb_upper", price * 1.05)
def r10(p): return int(round(p / 10) * 10)
# 손절선: 매입가 -8% 또는 MA60 -3% 중 높은 것
stop = max(r10(buy_price * 0.92), r10(ma60 * 0.97))
# 목표가
t1 = r10(max(buy_price * 1.08, bbu * 0.97)) # 본전+8% 또는 볼밴 상단
t2 = r10(max((price + h52) / 2, buy_price * 1.15))
t3 = r10(max(h52 * 0.97, buy_price * 1.25))
# 추가매수 구간 (물타기) - 현재가 -5%, -10%
avg_down1_price = r10(price * 0.95)
avg_down1_qty = max(1, qty // 3)
avg_down1_avg = (total_buy + avg_down1_price * avg_down1_qty) / (qty + avg_down1_qty)
avg_down2_price = r10(price * 0.90)
avg_down2_qty = max(1, qty // 2)
avg_down2_avg = (total_buy + avg_down2_price * avg_down2_qty) / (qty + avg_down2_qty)
return {
"pnl": pnl,
"pnl_pct": round(pnl_pct, 2),
"total_buy": total_buy,
"current_value": price * qty,
"stop_loss": stop,
"stop_pnl": (stop - buy_price) * qty,
"t1": t1, "t1_pnl": (t1 - buy_price) * qty,
"t2": t2, "t2_pnl": (t2 - buy_price) * qty,
"t3": t3, "t3_pnl": (t3 - buy_price) * qty,
"avg_down": [
{"price": avg_down1_price, "add_qty": avg_down1_qty,
"new_avg": round(avg_down1_avg), "add_cost": avg_down1_price * avg_down1_qty},
{"price": avg_down2_price, "add_qty": avg_down2_qty,
"new_avg": round(avg_down2_avg), "add_cost": avg_down2_price * avg_down2_qty},
],
"action": (
"손절 고려" if price <= stop else
"추가매수 검토" if pnl_pct < -5 and tech_score >= 0 else
"홀드" if -5 <= pnl_pct < 5 else
"1차 익절 고려" if pnl_pct >= 10 else "홀드"
),
}
# ── 단일 종목 분석 ────────────────────────────────────────
EXCLUDE_KEYWORDS = (
"기업인수목적", "선박투자회사", "부동산투자회사", "특별자산", "인프라투자",
"사모투자", "맥쿼리", "리츠", "REITs",
)
async def analyze_stock(client: httpx.AsyncClient, code: str, name: str = "",
with_ai: bool = False, news_score: float = 0) -> Optional[dict]:
# 이름이 없으면 DB에서 조회
if not name and pg_pool:
try:
async with pg_pool.acquire() as conn:
name = await conn.fetchval(
"SELECT corp_name FROM dart_corps WHERE stock_code=$1", code) or ""
except: pass
# SPAC·REITs·선박펀드 등 제외
if any(kw in name for kw in EXCLUDE_KEYWORDS):
return None
ohlcv = await get_ohlcv(client, code, 120)
if len(ohlcv) < 20:
return None
ind = calc_indicators(ohlcv)
if not ind:
return None
# 주가 500원 미만 penny stock 제외
if ind.get("price", 0) < 500:
return None
tech_score, signals = calc_tech_score(ind)
sig = "매수" if tech_score >= 30 else ("매도" if tech_score <= -30 else "관망")
# 관망도 목표가 계산 (기술점수 양수면 매수 기준, 음수면 매도 기준)
tgt_sig = "매수" if tech_score >= 0 else "매도"
targets = calc_price_targets(ind["price"], ind, tgt_sig)
ai_opinion = ""
if with_ai and sig != "관망":
ai_opinion = await generate_ai_opinion(
client, code, name, ind, tech_score, signals, targets, news_score)
result = {
"code": code, "name": name,
"tech_score": tech_score, "signal": sig,
"signals": signals, "indicators": ind, "targets": targets,
"ai_opinion": ai_opinion,
"analyzed_at": datetime.now().isoformat(),
}
if redis_cl:
try:
await redis_cl.set(f"ta:{code}", json.dumps(result, ensure_ascii=False), ex=1800)
except: pass
if pg_pool:
try:
async with pg_pool.acquire() as conn:
await conn.execute("""
INSERT INTO stock_technical (
stock_code, stock_name, price,
ma5, ma20, ma60, ma120,
rsi, macd, macd_signal, macd_hist,
bb_upper, bb_mid, bb_lower, pct_b,
stoch_k, stoch_d, vol_ratio,
tech_score, signal, signals, targets, analyzed_at
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23)
ON CONFLICT (stock_code) DO UPDATE SET
stock_name=$2, price=$3,
ma5=$4, ma20=$5, ma60=$6, ma120=$7,
rsi=$8, macd=$9, macd_signal=$10, macd_hist=$11,
bb_upper=$12, bb_mid=$13, bb_lower=$14, pct_b=$15,
stoch_k=$16, stoch_d=$17, vol_ratio=$18,
tech_score=$19, signal=$20, signals=$21, targets=$22, analyzed_at=$23
""",
code, name, ind["price"],
ind["ma5"], ind["ma20"], ind["ma60"], ind["ma120"],
ind["rsi"], ind["macd"], ind["macd_signal"], ind["macd_hist"],
ind["bb_upper"], ind["bb_mid"], ind["bb_lower"], ind["pct_b"],
ind["stoch_k"], ind["stoch_d"], ind["vol_ratio"],
tech_score, sig,
json.dumps(signals, ensure_ascii=False),
json.dumps(targets, ensure_ascii=False),
datetime.now())
except Exception as e:
logger.warning("ta.db.err", code=code, error=str(e))
return result
# ── DB 초기화 ─────────────────────────────────────────────
async def init_db():
async with pg_pool.acquire() as conn:
await conn.execute("""
CREATE TABLE IF NOT EXISTS stock_technical (
id SERIAL PRIMARY KEY,
stock_code VARCHAR(10) UNIQUE NOT NULL,
stock_name VARCHAR(100) DEFAULT '',
price INTEGER DEFAULT 0,
ma5 FLOAT DEFAULT 0,
ma20 FLOAT DEFAULT 0,
ma60 FLOAT DEFAULT 0,
ma120 FLOAT DEFAULT 0,
rsi FLOAT DEFAULT 50,
macd FLOAT DEFAULT 0,
macd_signal FLOAT DEFAULT 0,
macd_hist FLOAT DEFAULT 0,
bb_upper FLOAT DEFAULT 0,
bb_mid FLOAT DEFAULT 0,
bb_lower FLOAT DEFAULT 0,
pct_b FLOAT DEFAULT 0.5,
stoch_k FLOAT DEFAULT 50,
stoch_d FLOAT DEFAULT 50,
vol_ratio FLOAT DEFAULT 1,
tech_score FLOAT DEFAULT 0,
signal VARCHAR(10) DEFAULT '관망',
signals JSONB DEFAULT '[]'::jsonb,
targets JSONB DEFAULT '{}'::jsonb,
analyzed_at TIMESTAMP DEFAULT NOW()
)
""")
await conn.execute("CREATE INDEX IF NOT EXISTS idx_ta_score ON stock_technical(tech_score DESC)")
await conn.execute("CREATE INDEX IF NOT EXISTS idx_ta_signal ON stock_technical(signal)")
logger.info("ta.db.initialized")
# ── 전체 분석 작업 ────────────────────────────────────────
async def job_analyze():
logger.info("ta.job.start")
async with httpx.AsyncClient() as client:
codes: List[tuple] = []
if pg_pool:
try:
rows = await pg_pool.fetch("SELECT stock_code, corp_name FROM dart_corps LIMIT 500")
codes = [(r["stock_code"], r["corp_name"] or "") for r in rows if r["stock_code"]]
except: pass
if not codes:
for sosok in [0, 1]:
for page in range(1, 30):
try:
r = await client.get(
f"https://finance.naver.com/sise/sise_market_sum.naver?sosok={sosok}&page={page}",
headers=HEADERS, timeout=15)
r.encoding = "euc-kr"
found = re.findall(r'main\.naver\?code=(\d{6})[^>]*>([^<]+)</a>', r.text)
if not found: break
codes.extend([(c.strip(), n.strip()) for c, n in found])
await asyncio.sleep(0.2)
except: break
if len(codes) >= 500: break
ok = 0
for code, name in codes[:500]:
if not code or len(code) != 6: continue
try:
result = await analyze_stock(client, code, name)
if result: ok += 1
except Exception as e:
stats.errors += 1
logger.warning("ta.analyze.err", code=code, error=str(e))
await asyncio.sleep(0.4)
stats.analyzed += ok
stats.last_run = datetime.now().isoformat()
logger.info("ta.job.done", analyzed=ok)
# ── FastAPI ────────────────────────────────────────────────
app = FastAPI(title="기술적 분석 엔진")
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
@app.on_event("startup")
async def startup():
global pg_pool, redis_cl
pg_pool = await asyncpg.create_pool(
host=PG_HOST, port=PG_PORT, database=PG_DB,
user=PG_USER, password=PG_PASS, min_size=2, max_size=5)
redis_cl = aioredis.Redis(
host=REDIS_HOST, port=6379, password=REDIS_PASSWORD, db=5, decode_responses=True)
await init_db()
scheduler.add_job(job_analyze, "cron", day_of_week="mon-fri",
hour="9-16", minute="*/30", id="ta_30m", replace_existing=True)
scheduler.add_job(job_analyze, "cron", day_of_week="mon-fri",
hour=16, minute=15, id="ta_close", replace_existing=True)
scheduler.start()
logger.info("ta-engine.started")
@app.on_event("shutdown")
async def shutdown():
scheduler.shutdown()
if pg_pool: await pg_pool.close()
if redis_cl: await redis_cl.aclose()
@app.get("/health")
async def health():
return {"status": "ok", "analyzed": stats.analyzed,
"errors": stats.errors, "last_run": stats.last_run}
@app.get("/technical/{code}")
async def technical(code: str):
if redis_cl:
cached = await redis_cl.get(f"ta:{code}")
if cached:
return JSONResponse(content=json.loads(cached))
if pg_pool:
async with pg_pool.acquire() as conn:
row = await conn.fetchrow("SELECT * FROM stock_technical WHERE stock_code=$1", code)
if row:
d = dict(row)
d["analyzed_at"] = str(d["analyzed_at"])
for k in ("signals", "targets"):
if isinstance(d[k], str):
d[k] = json.loads(d[k])
return JSONResponse(content=d)
# 실시간 분석
async with httpx.AsyncClient() as client:
result = await analyze_stock(client, code)
if result:
return JSONResponse(content=result)
return JSONResponse(content={"error": "not found"}, status_code=404)
@app.get("/ranking")
async def ranking(limit: int = Query(default=30), signal: str = Query(default="")):
async with pg_pool.acquire() as conn:
if signal:
rows = await conn.fetch(
"SELECT * FROM stock_technical WHERE signal=$1 ORDER BY tech_score DESC LIMIT $2",
signal, limit)
else:
rows = await conn.fetch(
"SELECT * FROM stock_technical ORDER BY tech_score DESC LIMIT $1", limit)
result = []
for row in rows:
d = dict(row)
d["analyzed_at"] = str(d["analyzed_at"])
for k in ("signals", "targets"):
if isinstance(d[k], str):
d[k] = json.loads(d[k])
result.append(d)
return result
@app.get("/buy-candidates")
async def buy_candidates(limit: int = Query(default=20)):
"""기술적 매수 후보 (점수 30 이상) + 펀더멘탈 점수 합산"""
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT t.*,
s.news_score, s.dart_score, s.recommendation AS fundamental_rec,
s.total_score AS fundamental_total
FROM stock_technical t
LEFT JOIN stock_scores s
ON t.stock_code = s.stock_code
AND s.score_date = (SELECT MAX(score_date) FROM stock_scores)
WHERE t.signal = '매수' AND t.tech_score >= 30
ORDER BY (t.tech_score + COALESCE(s.total_score, 0)) DESC
LIMIT $1
""", limit)
result = []
for row in rows:
d = dict(row)
d["analyzed_at"] = str(d["analyzed_at"])
for k in ("signals", "targets"):
if isinstance(d.get(k), str):
d[k] = json.loads(d[k])
result.append(d)
return result
@app.post("/analyze/all")
async def analyze_all():
asyncio.create_task(job_analyze())
return {"status": "started"}
@app.post("/analyze/{code}")
async def analyze_single(code: str, ai: bool = False):
async with httpx.AsyncClient() as client:
result = await analyze_stock(client, code, with_ai=ai)
if result:
return JSONResponse(content=result)
return JSONResponse(content={"error": "analysis failed"}, status_code=500)
# ── 보유 포지션 맞춤 분석 ────────────────────────────────
@app.post("/position")
async def position_analysis(req: PositionRequest, ai: bool = False):
"""보유 종목 매입가/수량 기반 맞춤 손익 + 전략 분석"""
code = req.code
# 캐시 확인
result = None
if redis_cl:
try:
cached = await redis_cl.get(f"ta:{code}")
if cached:
result = json.loads(cached)
except: pass
if not result:
async with httpx.AsyncClient() as client:
result = await analyze_stock(client, code, req.name, with_ai=False)
if not result:
return JSONResponse(content={"error": "종목 분석 실패"}, status_code=500)
ind = result.get("indicators", {})
tech_score = result.get("tech_score", 0)
price = ind.get("price", 0)
pos = analyze_position(price, req.buy_price, req.qty, ind, tech_score)
# AI 판단문 (요청 시)
ai_opinion = result.get("ai_opinion", "")
if ai and not ai_opinion:
async with httpx.AsyncClient() as client:
ai_opinion = await generate_ai_opinion(
client, code, req.name or result.get("name", code),
ind, tech_score, result.get("signals", []),
result.get("targets", {}))
return {
"code": code,
"name": req.name or result.get("name", code),
"buy_price": req.buy_price,
"qty": req.qty,
"current_price": price,
"tech_score": tech_score,
"signal": result.get("signal"),
"signals": result.get("signals", []),
"indicators": ind,
"position": pos,
"targets": result.get("targets", {}),
"ai_opinion": ai_opinion,
"analyzed_at": result.get("analyzed_at"),
}
# ── 종목 전체 리포트 (AI 판단문 포함) ────────────────────
@app.get("/report/{code}")
async def full_report(code: str):
"""기술적 분석 + AI 판단문 + 뉴스감성 통합 리포트"""
news_score = 0.0
if pg_pool:
try:
async with pg_pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT news_score FROM stock_scores WHERE stock_code=$1 "
"ORDER BY score_date DESC LIMIT 1", code)
if row:
news_score = float(row["news_score"] or 0)
except: pass
async with httpx.AsyncClient() as client:
result = await analyze_stock(client, code, with_ai=True, news_score=news_score)
if not result:
return JSONResponse(content={"error": "분석 실패"}, status_code=500)
# DB에서 추가 정보
extra = {}
if pg_pool:
try:
async with pg_pool.acquire() as conn:
score_row = await conn.fetchrow(
"SELECT * FROM stock_scores WHERE stock_code=$1 "
"ORDER BY score_date DESC LIMIT 1", code)
news_rows = await conn.fetch(
"SELECT title, sentiment, intensity, reason "
"FROM news_analysis WHERE primary_stock=$1 "
"ORDER BY analyzed_at DESC LIMIT 5", code)
if score_row:
extra["score"] = dict(score_row)
extra["score"]["score_date"] = str(extra["score"]["score_date"])
extra["recent_news"] = [dict(r) for r in news_rows]
except: pass
return {**result, **extra, "news_score": news_score}
+10
View File
@@ -0,0 +1,10 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
httpx==0.27.0
redis==5.0.4
asyncpg==0.29.0
apscheduler==3.10.4
orjson==3.10.3
structlog==24.2.0
yfinance==0.2.54
pydantic==2.7.1
+7
View File
@@ -0,0 +1,7 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "-u", "main.py"]
+499
View File
@@ -0,0 +1,499 @@
"""
Trading AI Telegram Bot (양방향 대화)
명령어:
/start, /help 안내
/추천 또는 /buy 오늘 강력매수 top 5
/매도 또는 /sell 오늘 강력매도 top 3
/종목 005930 또는 /stock 005930 특정 종목 상세
/시장 또는 /market 시장 상황 요약
자유 텍스트 EXAONE 3.5 시스템 DB 데이터 + 사용자 질문 종합해 답변
"""
import os
import re
import asyncio
import json
import logging
from datetime import datetime
from typing import Optional, List
import asyncpg
import httpx
import structlog
from telegram import Update, BotCommand
from telegram.constants import ParseMode
from telegram.ext import (Application, CommandHandler, MessageHandler,
filters, ContextTypes)
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s")
logger = structlog.get_logger()
PG = {
"host": os.getenv("POSTGRES_HOST", "postgres"),
"port": int(os.getenv("POSTGRES_PORT", 5432)),
"database": os.getenv("POSTGRES_DB", "trading_ai"),
"user": os.getenv("POSTGRES_USER", "kyu"),
"password": os.getenv("POSTGRES_PASSWORD", ""),
}
TG_TOKEN = os.getenv("TELEGRAM_BOT_TOKEN", "")
ALLOWED_CHAT_ID = os.getenv("TELEGRAM_CHAT_ID", "") # 허용된 사용자만
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://ollama:11434")
EXAONE_MODEL = os.getenv("EXAONE_MODEL", "exaone3.5:7.8b")
SCORE_API = os.getenv("SCORE_API_URL", "http://score-engine:8686")
pg_pool: Optional[asyncpg.Pool] = None
# ─────────────────────────────────────────────────────────────
# 권한 체크
# ─────────────────────────────────────────────────────────────
def is_allowed(update: Update) -> bool:
if not ALLOWED_CHAT_ID:
return True
return str(update.effective_chat.id) == ALLOWED_CHAT_ID
# ─────────────────────────────────────────────────────────────
# DB 조회 헬퍼
# ─────────────────────────────────────────────────────────────
async def db_top_buys(limit: int = 5) -> list:
async with pg_pool.acquire() as conn:
return await conn.fetch("""
SELECT s.stock_code, s.stock_name, s.total_score, s.recommendation,
s.us_overnight_adj, s.us_pair_top,
s.insider_signal, s.consensus_signal, s.macro_signal,
s.inst_flow_signal, s.valuation_pct,
s.top_reasons
FROM stock_scores s
WHERE s.score_date=(SELECT MAX(score_date) FROM stock_scores)
AND s.recommendation IN ('강력매수','매수관심')
ORDER BY s.total_score DESC LIMIT $1
""", limit)
async def db_top_sells(limit: int = 3) -> list:
async with pg_pool.acquire() as conn:
return await conn.fetch("""
SELECT stock_code, stock_name, total_score, recommendation, top_reasons
FROM stock_scores
WHERE score_date=(SELECT MAX(score_date) FROM stock_scores)
AND recommendation IN ('강력매도','매도관심')
ORDER BY total_score ASC LIMIT $1
""", limit)
async def db_stock_detail(code: str) -> Optional[dict]:
async with pg_pool.acquire() as conn:
row = await conn.fetchrow("""
SELECT s.*,
d.corp_name, d.sector_name,
f.roe, f.operating_margin, f.debt_ratio, f.revenue_growth,
f.fcf_ratio
FROM stock_scores s
JOIN dart_corps d ON d.stock_code=s.stock_code
LEFT JOIN dart_financials f ON f.stock_code=s.stock_code
AND f.reprt_code='11011'
AND f.bsns_year=(SELECT MAX(bsns_year) FROM dart_financials f2
WHERE f2.stock_code=s.stock_code AND f2.reprt_code='11011')
WHERE s.stock_code=$1
AND s.score_date=(SELECT MAX(score_date) FROM stock_scores
WHERE stock_code=$1)
""", code)
return dict(row) if row else None
async def db_market_regime() -> Optional[dict]:
async with pg_pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT regime, regime_adj, dt FROM market_regime ORDER BY dt DESC LIMIT 1")
macro = await conn.fetch("""
SELECT DISTINCT ON (indicator) indicator, trade_date, value
FROM macro_daily ORDER BY indicator, trade_date DESC
""")
return {
"regime": dict(row) if row else None,
"macro": {r["indicator"]: float(r["value"]) for r in macro}
}
async def find_stock_code_by_name(name: str) -> Optional[str]:
"""이름으로 종목코드 검색"""
async with pg_pool.acquire() as conn:
row = await conn.fetchrow("""
SELECT stock_code FROM dart_corps
WHERE is_active=true AND corp_name ILIKE $1
ORDER BY CASE WHEN corp_name=$2 THEN 0 ELSE 1 END
LIMIT 1
""", f"%{name}%", name)
return row["stock_code"] if row else None
# ─────────────────────────────────────────────────────────────
# EXAONE 호출
# ─────────────────────────────────────────────────────────────
SYSTEM_PROMPT = """당신은 워렌 버핏 스타일의 한국 주식 가치투자 분석가입니다.
사용자 질문에 대해 제공된 시스템 데이터를 우선 참고하여 답변하세요.
원칙:
- 데이터에 근거한 객관적 답변
- 짧고 명확하게 (3~5문장)
- ROE/부채비율/영업이익률 중심의 가치투자 관점
- 투자는 본인 책임이라는 마지막에 짧게 언급
- 데이터에 없는 내용은 추측하지 말고 "모르겠습니다"라고 말할
"""
async def ask_exaone(user_msg: str, context: str = "") -> str:
prompt = f"{SYSTEM_PROMPT}\n\n=== 시스템 데이터 ===\n{context}\n\n=== 사용자 질문 ===\n{user_msg}"
try:
async with httpx.AsyncClient(timeout=120) as cl:
r = await cl.post(f"{OLLAMA_URL}/api/generate", json={
"model": EXAONE_MODEL, "prompt": prompt, "stream": False,
"options": {"temperature": 0.3, "num_predict": 400}
})
if r.status_code != 200:
return f"⚠️ LLM 호출 실패 (status {r.status_code})"
j = r.json()
return j.get("response", "").strip() or "(빈 응답)"
except Exception as e:
return f"⚠️ LLM 오류: {e}"
# ─────────────────────────────────────────────────────────────
# 포매팅 헬퍼
# ─────────────────────────────────────────────────────────────
def fmt_pct(v) -> str:
try: return f"{float(v):+.1f}"
except: return "-"
def fmt_buys(rows: list) -> str:
if not rows:
return "📭 오늘 매수 추천 없음"
lines = ["🟢 <b>오늘 매수 추천</b>"]
for i, r in enumerate(rows, 1):
emoji = "🔥" if r["recommendation"] == "강력매수" else ""
line = f"{emoji} <b>{i}. {r['stock_name']}({r['stock_code']})</b> {r['total_score']:.1f}"
lines.append(line)
sigs = []
if r["us_overnight_adj"] and r["us_overnight_adj"] != 0:
sigs.append(f"US{fmt_pct(r['us_overnight_adj'])}")
if r["insider_signal"] and r["insider_signal"] != 0:
sigs.append(f"내부자{fmt_pct(r['insider_signal'])}")
if r["consensus_signal"] and r["consensus_signal"] != 0:
sigs.append(f"컨센{fmt_pct(r['consensus_signal'])}")
if r["valuation_pct"] and r["valuation_pct"] != 0:
sigs.append(f"밸류{fmt_pct(r['valuation_pct'])}")
if r["inst_flow_signal"] and r["inst_flow_signal"] != 0:
sigs.append(f"기관{fmt_pct(r['inst_flow_signal'])}")
if sigs:
lines.append(f"{' / '.join(sigs)}")
if r["top_reasons"]:
lines.append(f"{r['top_reasons'][:80]}")
return "\n".join(lines)
def fmt_sells(rows: list) -> str:
if not rows:
return "📭 오늘 매도 추천 없음"
lines = ["🔴 <b>오늘 매도 신호</b>"]
for i, r in enumerate(rows, 1):
lines.append(f"<b>{i}. {r['stock_name']}({r['stock_code']})</b> {r['total_score']:.1f}점 · {r['recommendation']}")
if r["top_reasons"]:
lines.append(f"{r['top_reasons'][:80]}")
return "\n".join(lines)
def fmt_stock(d: dict) -> str:
if not d:
return "❓ 종목을 찾을 수 없습니다"
lines = [
f"📊 <b>{d['corp_name']}({d['stock_code']})</b>",
f"종합점수 <b>{d['total_score']:.1f}</b> · {d['recommendation']}",
f"섹터: {d.get('sector_name') or '미분류'}",
"",
"<b>📈 시그널</b>",
]
sigs = [
("뉴스", d.get("news_score")),
("공시", d.get("dart_score")),
("기술", d.get("technical_score")),
("외국인", d.get("foreign_score")),
("공매도", d.get("short_score")),
("미국동조", d.get("us_overnight_adj")),
("내부자", d.get("insider_signal")),
("컨센", d.get("consensus_signal")),
("매크로", d.get("macro_signal")),
("기관 5d", d.get("inst_flow_signal")),
("밸류", d.get("valuation_pct")),
]
for k, v in sigs:
if v and float(v) != 0:
lines.append(f" {k}: {fmt_pct(v)}")
lines.append("")
lines.append("<b>💰 재무</b>")
if d.get("roe") is not None:
lines.append(f" ROE {d['roe']:.1f}% · 부채 {d.get('debt_ratio',0):.0f}% · 영업이익률 {d.get('operating_margin',0):.1f}%")
if d.get("intrinsic_value"):
lines.append(f" 내재가치 {int(d['intrinsic_value']):,}원 · 안전마진 {d.get('margin_of_safety',0):.0f}%")
if d.get("us_pair_top"):
lines.append("")
lines.append(f"🇺🇸 {d['us_pair_top']}")
if d.get("top_reasons"):
lines.append("")
lines.append(f"<i>{d['top_reasons'][:200]}</i>")
return "\n".join(lines)
# ─────────────────────────────────────────────────────────────
# 핸들러
# ─────────────────────────────────────────────────────────────
async def cmd_start(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
if not is_allowed(update): return
msg = (
"👋 <b>Trading AI 봇</b>\n\n"
"명령어:\n"
"/buy — 오늘 매수 추천\n"
"/sell — 오늘 매도 신호\n"
"/stock 005930 — 특정 종목 상세\n"
"/deep 005930 — AI 심층분석(RAG+EXAONE)\n"
"/market — 시장 상황\n"
"/help — 도움말\n\n"
"자유 텍스트로 질문해도 됩니다. 예:\n"
"<i>\"에스엠 사도 돼?\"</i>\n"
"<i>\"오늘 추천 알려줘\"</i>\n"
"<i>\"삼성전자 어때?\"</i>"
)
await update.message.reply_text(msg, parse_mode=ParseMode.HTML)
async def cmd_help(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
await cmd_start(update, ctx)
async def cmd_buys(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
if not is_allowed(update): return
rows = await db_top_buys(10)
await update.message.reply_text(fmt_buys(rows), parse_mode=ParseMode.HTML)
async def cmd_sells(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
if not is_allowed(update): return
rows = await db_top_sells(5)
await update.message.reply_text(fmt_sells(rows), parse_mode=ParseMode.HTML)
async def cmd_stock(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
if not is_allowed(update): return
args = ctx.args
if not args:
await update.message.reply_text("종목코드 또는 이름을 입력하세요. 예: /종목 005930 또는 /종목 삼성전자")
return
q = " ".join(args).strip()
code = q if re.match(r"^\d{6}$", q) else await find_stock_code_by_name(q)
if not code:
await update.message.reply_text(f"'{q}' 종목을 찾을 수 없습니다")
return
d = await db_stock_detail(code)
await update.message.reply_text(fmt_stock(d), parse_mode=ParseMode.HTML)
def fmt_deep(r: dict) -> str:
if r.get("error"):
return f"⚠️ {r.get('code')}: {r['error']}"
icon = {"강력매수": "🟢🟢", "매수": "🟢", "중립": "",
"매도": "🔴", "강력매도": "🔴🔴"}.get(r.get("recommendation"), "")
lines = [
f"{icon} <b>{r.get('name')}({r.get('code')})</b> — AI심층분석",
f"판단: <b>{r.get('recommendation')}</b> (확신도 {r.get('conviction')}/5) "
f"· 퀀트 {r.get('quant_score',0):.0f}",
f"밸류: {r.get('valuation_view','-')} · 기간: {r.get('time_horizon','-')}",
"",
f"<b>📝 투자논거</b>\n{r.get('thesis','')}",
]
if r.get("key_points"):
lines.append("\n<b>✅ 핵심근거</b>")
lines += [f"{p}" for p in r["key_points"][:5]]
if r.get("risks"):
lines.append("\n<b>⚠️ 리스크</b>")
lines += [f"{p}" for p in r["risks"][:4]]
if r.get("catalyst_watch"):
lines.append("\n<b>👀 관전포인트</b>")
lines += [f"{p}" for p in r["catalyst_watch"][:3]]
tp, sl = r.get("target_price", 0) or 0, r.get("stop_loss", 0) or 0
if tp or sl:
lines.append(f"\n🎯 목표가 {tp:,}원 · 손절가 {sl:,}")
lines.append("\n<i>※ 투자 판단·책임은 본인에게 있습니다</i>")
return "\n".join(lines)
async def cmd_deep(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
if not is_allowed(update): return
args = ctx.args
if not args:
await update.message.reply_text("종목코드/이름 입력. 예: /deep 005930 또는 /deep 삼성전자")
return
q = " ".join(args).strip()
code = q if re.match(r"^\d{6}$", q) else await find_stock_code_by_name(q)
if not code:
await update.message.reply_text(f"'{q}' 종목을 찾을 수 없습니다")
return
await update.message.reply_text("🧠 RAG+EXAONE 심층분석 중… (최대 1~2분)")
try:
async with httpx.AsyncClient(timeout=200) as cl:
resp = await cl.get(f"{SCORE_API}/deep-analysis/{code}",
params={"refresh": "true", "notify": "false"})
r = resp.json()
except Exception as e:
await update.message.reply_text(f"⚠️ 분석 실패: {e}")
return
await update.message.reply_text(fmt_deep(r), parse_mode=ParseMode.HTML)
async def cmd_market(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
if not is_allowed(update): return
m = await db_market_regime()
regime = m.get("regime") or {}
macro = m.get("macro") or {}
lines = ["🌍 <b>시장 상황</b>"]
if regime:
lines.append(f" 레짐: <b>{regime.get('regime','?')}</b> (보정 {regime.get('regime_adj',0):+.0f})")
lines.append(f" 날짜: {regime.get('dt')}")
lines.append("")
lines.append("<b>매크로</b>")
if "usdkrw" in macro: lines.append(f" USD/KRW {macro['usdkrw']:.1f}")
if "kor_10y" in macro: lines.append(f" 국고채 10년 {macro['kor_10y']:.2f}%")
if "kor_3y" in macro: lines.append(f" 국고채 3년 {macro['kor_3y']:.2f}%")
if "kospi" in macro: lines.append(f" KOSPI {macro['kospi']:,.1f}")
await update.message.reply_text("\n".join(lines), parse_mode=ParseMode.HTML)
# 자유 텍스트
async def on_text(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
if not is_allowed(update): return
text = (update.message.text or "").strip()
if not text:
return
# 한글 단축 키워드 → 명령 라우팅 (빠른 응답)
if text in ("추천", "매수", "오늘추천", "buy"):
rows = await db_top_buys(10)
await update.message.reply_text(fmt_buys(rows), parse_mode=ParseMode.HTML)
return
if text in ("매도", "팔것", "sell"):
rows = await db_top_sells(5)
await update.message.reply_text(fmt_sells(rows), parse_mode=ParseMode.HTML)
return
if text in ("시장", "", "market"):
await cmd_market(update, ctx)
return
await update.message.chat.send_action("typing")
# 컨텍스트 구성: 종목명 언급되면 그 종목 데이터, 아니면 시장 전반
context_parts = []
# 종목코드 / 종목명 감지
code = None
code_match = re.search(r"\b(\d{6})\b", text)
if code_match:
code = code_match.group(1)
if not code:
# 종목명 추정 (2~10자 한글)
for cand in re.findall(r"[가-힣A-Z]{2,10}", text):
if cand in ("매수", "매도", "추천", "오늘", "지금", "사도", "팔아", "어때"):
continue
c = await find_stock_code_by_name(cand)
if c:
code = c
break
if code:
d = await db_stock_detail(code)
if d:
context_parts.append(f"[종목 {d['corp_name']}({code}) 데이터]")
context_parts.append(f"종합점수 {d.get('total_score',0):.1f}점, 등급 {d.get('recommendation','-')}, 섹터 {d.get('sector_name') or '미분류'}")
context_parts.append(
f"세부 점수 — 뉴스 {d.get('news_score',0):.0f}, 기술 {d.get('technical_score',0):.0f}, "
f"공시 {d.get('dart_score',0):.0f}, 외국인 {d.get('foreign_score',0):.0f}, "
f"미국동조 {d.get('us_overnight_adj',0):.1f}, 내부자 {d.get('insider_signal',0):.1f}, "
f"컨센서스 {d.get('consensus_signal',0):.1f}, 기관5d {d.get('inst_flow_signal',0):.1f}, "
f"밸류 percentile {d.get('valuation_pct',0):.1f}")
if d.get("roe") is not None:
context_parts.append(
f"재무 — ROE {d['roe']:.1f}%, 영업이익률 {d.get('operating_margin',0):.1f}%, "
f"부채비율 {d.get('debt_ratio',0):.0f}%, 매출성장 {d.get('revenue_growth',0):.1f}%")
if d.get("intrinsic_value"):
context_parts.append(
f"DCF 내재가치 {int(d['intrinsic_value']):,}원, 안전마진 {d.get('margin_of_safety',0):.0f}%")
if d.get("us_pair_top"):
context_parts.append(f"미국 페어: {d['us_pair_top']}")
if d.get("top_reasons"):
context_parts.append(f"근거: {d['top_reasons']}")
else:
# 시장 전반
buys = await db_top_buys(5)
sells = await db_top_sells(3)
m = await db_market_regime()
regime = (m.get("regime") or {})
macro = m.get("macro") or {}
context_parts.append(f"[시장 상황] 레짐: {regime.get('regime','?')}")
if macro:
context_parts.append(
f"환율 {macro.get('usdkrw',0):.0f}원, 10년물 {macro.get('kor_10y',0):.2f}%, "
f"KOSPI {macro.get('kospi',0):,.0f}")
if buys:
context_parts.append("[오늘 매수 추천 top]")
for r in buys[:5]:
context_parts.append(f"- {r['stock_name']}({r['stock_code']}) {r['total_score']:.0f}{r['recommendation']}")
if sells:
context_parts.append("[오늘 매도 신호]")
for r in sells[:3]:
context_parts.append(f"- {r['stock_name']}({r['stock_code']}) {r['total_score']:.0f}{r['recommendation']}")
context_str = "\n".join(context_parts) if context_parts else "(관련 데이터 없음)"
answer = await ask_exaone(text, context_str)
# 답변 길이 제한
if len(answer) > 3500:
answer = answer[:3500] + "..."
await update.message.reply_text(answer)
# ─────────────────────────────────────────────────────────────
# 부트
# ─────────────────────────────────────────────────────────────
async def post_init(app: Application):
global pg_pool
pg_pool = await asyncpg.create_pool(**PG, min_size=2, max_size=5)
# 커맨드 메뉴 등록
await app.bot.set_my_commands([
BotCommand("buy", "오늘 매수 추천"),
BotCommand("sell", "오늘 매도 신호"),
BotCommand("stock", "종목 상세 (예: /stock 005930)"),
BotCommand("market", "시장 상황"),
BotCommand("help", "도움말"),
])
logger.info("telegram-bot.ready")
def main():
if not TG_TOKEN:
logger.error("missing TELEGRAM_BOT_TOKEN")
return
app = Application.builder().token(TG_TOKEN).post_init(post_init).build()
# 한국어/영어 명령어 둘 다
app.add_handler(CommandHandler("start", cmd_start))
app.add_handler(CommandHandler("help", cmd_help))
app.add_handler(CommandHandler(["buy", "buys"], cmd_buys))
app.add_handler(CommandHandler(["sell", "sells"], cmd_sells))
app.add_handler(CommandHandler(["stock"], cmd_stock))
app.add_handler(CommandHandler(["deep"], cmd_deep))
app.add_handler(CommandHandler(["market"], cmd_market))
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, on_text))
logger.info("telegram-bot.start", model=EXAONE_MODEL)
app.run_polling(close_loop=False)
if __name__ == "__main__":
main()
+5
View File
@@ -0,0 +1,5 @@
python-telegram-bot[asyncio]==21.5
asyncpg==0.29.0
httpx==0.27.0
structlog==24.2.0
orjson==3.10.3
+268
View File
@@ -0,0 +1,268 @@
import { useState, useEffect, useCallback } from "react";
const API = "http://192.168.0.60:8989/api";
const SENTIMENT_MAP = { "호재": { color: "#00E676", icon: "▲", bg: "rgba(0,230,118,0.08)" }, "악재": { color: "#FF5252", icon: "▼", bg: "rgba(255,82,82,0.08)" }, "중립": { color: "#78909C", icon: "●", bg: "rgba(120,144,156,0.06)" } };
const REC_MAP = { "강력매수": { color: "#00E676", weight: 900 }, "매수관심": { color: "#69F0AE", weight: 700 }, "관망": { color: "#78909C", weight: 500 }, "매도관심": { color: "#FF8A80", weight: 700 }, "강력매도": { color: "#FF5252", weight: 900 } };
function useFetch(endpoint, interval = 60000) {
const [data, setData] = useState(null);
const [loading, setLoading] = useState(true);
const load = useCallback(() => {
fetch(`${API}${endpoint}`).then(r => r.json()).then(d => { setData(d); setLoading(false); }).catch(() => setLoading(false));
}, [endpoint]);
useEffect(() => { load(); const t = setInterval(load, interval); return () => clearInterval(t); }, [load, interval]);
return { data, loading, reload: load };
}
function formatTime(ts) {
if (!ts) return "-";
const d = new Date(ts);
const now = new Date();
const diff = (now - d) / 1000 / 60;
if (diff < 60) return `${Math.floor(diff)}분 전`;
if (diff < 1440) return `${Math.floor(diff / 60)}시간 전`;
return d.toLocaleDateString("ko-KR", { month: "short", day: "numeric" });
}
function ScoreBar({ value, max = 100 }) {
const pct = Math.min(100, Math.max(0, ((value + max) / (2 * max)) * 100));
const color = value >= 30 ? "#00E676" : value >= -30 ? "#78909C" : "#FF5252";
return (
<div style={{ width: "100%", height: 6, background: "rgba(255,255,255,0.05)", borderRadius: 3, overflow: "hidden" }}>
<div style={{ width: `${pct}%`, height: "100%", background: color, borderRadius: 3, transition: "width 0.8s ease" }} />
</div>
);
}
function SummaryCard({ icon, label, value, sub, color }) {
return (
<div style={{ background: "rgba(255,255,255,0.03)", border: "1px solid rgba(255,255,255,0.06)", borderRadius: 16, padding: "24px 20px", display: "flex", flexDirection: "column", gap: 8 }}>
<div style={{ fontSize: 13, color: "#90A4AE", letterSpacing: 1 }}>{icon} {label}</div>
<div style={{ fontSize: 36, fontWeight: 800, color: color || "#E0E0E0", fontFamily: "'JetBrains Mono', monospace", lineHeight: 1 }}>{value}</div>
{sub && <div style={{ fontSize: 12, color: "#607D8B" }}>{sub}</div>}
</div>
);
}
function Header({ onRefresh }) {
const [time, setTime] = useState(new Date());
useEffect(() => { const t = setInterval(() => setTime(new Date()), 1000); return () => clearInterval(t); }, []);
return (
<div style={{ display: "flex", justifyContent: "space-between", alignItems: "center", padding: "20px 0", borderBottom: "1px solid rgba(255,255,255,0.06)", marginBottom: 24 }}>
<div>
<h1 style={{ margin: 0, fontSize: 22, fontWeight: 800, color: "#E0E0E0", letterSpacing: -0.5 }}>
<span style={{ color: "#00E676" }}></span> Trading AI Dashboard
</h1>
<div style={{ fontSize: 12, color: "#546E7A", marginTop: 4 }}>뉴스 + DART 공시 기반 종목 분석 시스템</div>
</div>
<div style={{ display: "flex", alignItems: "center", gap: 16 }}>
<div style={{ fontSize: 13, color: "#607D8B", fontFamily: "'JetBrains Mono', monospace" }}>
{time.toLocaleString("ko-KR", { hour: "2-digit", minute: "2-digit", second: "2-digit" })}
</div>
<button onClick={onRefresh} style={{ background: "rgba(0,230,118,0.1)", border: "1px solid rgba(0,230,118,0.2)", borderRadius: 8, padding: "8px 16px", color: "#00E676", fontSize: 12, cursor: "pointer", fontWeight: 600 }}>
새로고침
</button>
</div>
</div>
);
}
function RecommendationCard({ item }) {
const rec = REC_MAP[item.recommendation] || REC_MAP["관망"];
return (
<div style={{ background: "rgba(255,255,255,0.02)", border: "1px solid rgba(255,255,255,0.05)", borderRadius: 12, padding: 16, display: "flex", flexDirection: "column", gap: 10 }}>
<div style={{ display: "flex", justifyContent: "space-between", alignItems: "flex-start" }}>
<div>
<div style={{ fontSize: 15, fontWeight: 700, color: "#E0E0E0" }}>{item.stock_name}</div>
<div style={{ fontSize: 11, color: "#546E7A", fontFamily: "'JetBrains Mono', monospace" }}>{item.stock_code}</div>
</div>
<div style={{ background: `${rec.color}15`, border: `1px solid ${rec.color}30`, borderRadius: 6, padding: "4px 10px", fontSize: 12, fontWeight: rec.weight, color: rec.color }}>
{item.recommendation}
</div>
</div>
<div style={{ display: "flex", gap: 12, fontSize: 11, color: "#78909C" }}>
<span>종합 <b style={{ color: "#E0E0E0", fontSize: 14 }}>{item.total_score?.toFixed(1)}</b></span>
<span>뉴스 <b style={{ color: "#69F0AE" }}>{item.news_score?.toFixed(0)}</b></span>
<span>공시 <b style={{ color: "#40C4FF" }}>{item.dart_score?.toFixed(0)}</b></span>
<span>가격 <b style={{ color: "#FFD740" }}>{item.price_score?.toFixed(0)}</b></span>
</div>
<ScoreBar value={item.total_score} />
{item.top_reasons && (
<div style={{ fontSize: 11, color: "#607D8B", lineHeight: 1.5, borderTop: "1px solid rgba(255,255,255,0.04)", paddingTop: 8 }}>
{item.top_reasons.split("|")[0]?.trim().substring(0, 120)}...
</div>
)}
</div>
);
}
function NewsItem({ item }) {
const s = SENTIMENT_MAP[item.sentiment] || SENTIMENT_MAP["중립"];
return (
<div style={{ display: "flex", gap: 12, padding: "12px 0", borderBottom: "1px solid rgba(255,255,255,0.03)" }}>
<div style={{ minWidth: 40, textAlign: "center" }}>
<div style={{ fontSize: 18, color: s.color }}>{s.icon}</div>
<div style={{ fontSize: 10, color: s.color, fontWeight: 700 }}>{item.intensity || 0}</div>
</div>
<div style={{ flex: 1, minWidth: 0 }}>
<div style={{ fontSize: 13, color: "#CFD8DC", lineHeight: 1.4, overflow: "hidden", textOverflow: "ellipsis", whiteSpace: "nowrap" }}>
{item.title}
</div>
<div style={{ display: "flex", gap: 8, marginTop: 4, fontSize: 11 }}>
{item.primary_stock && <span style={{ color: "#40C4FF" }}>{item.primary_stock}</span>}
<span style={{ color: "#546E7A" }}>{item.source}</span>
<span style={{ color: "#455A64" }}>{formatTime(item.analyzed_at)}</span>
</div>
{item.reason && <div style={{ fontSize: 11, color: "#607D8B", marginTop: 4, lineHeight: 1.4 }}>{item.reason.substring(0, 100)}</div>}
</div>
<div style={{ background: s.bg, borderRadius: 6, padding: "4px 8px", fontSize: 11, color: s.color, fontWeight: 600, height: "fit-content", whiteSpace: "nowrap" }}>
{item.investment_action || "관망"}
</div>
</div>
);
}
function AlertItem({ item }) {
const s = SENTIMENT_MAP[item.sentiment] || SENTIMENT_MAP["중립"];
return (
<div style={{ background: `${s.color}08`, border: `1px solid ${s.color}20`, borderRadius: 10, padding: 12, marginBottom: 8 }}>
<div style={{ display: "flex", justifyContent: "space-between", alignItems: "center" }}>
<div style={{ fontSize: 13, fontWeight: 600, color: s.color }}>
{s.icon} {item.sentiment} (강도 {item.intensity})
</div>
<div style={{ fontSize: 10, color: "#546E7A" }}>{formatTime(item.analyzed_at)}</div>
</div>
<div style={{ fontSize: 12, color: "#CFD8DC", marginTop: 6, lineHeight: 1.4 }}>{item.title}</div>
<div style={{ fontSize: 11, color: "#78909C", marginTop: 4 }}>{item.reason?.substring(0, 100)}</div>
</div>
);
}
function Tab({ active, label, onClick }) {
return (
<button onClick={onClick} style={{
background: active ? "rgba(0,230,118,0.1)" : "transparent",
border: active ? "1px solid rgba(0,230,118,0.2)" : "1px solid transparent",
borderRadius: 8, padding: "8px 16px", color: active ? "#00E676" : "#607D8B",
fontSize: 13, fontWeight: active ? 700 : 500, cursor: "pointer", transition: "all 0.2s"
}}>{label}</button>
);
}
export default function Dashboard() {
const [tab, setTab] = useState("overview");
const { data: summary, reload: r1 } = useFetch("/summary", 30000);
const { data: ranking } = useFetch("/ranking", 60000);
const { data: recs, reload: r2 } = useFetch("/recommendations", 60000);
const { data: recent } = useFetch("/recent?limit=30", 30000);
const { data: alerts } = useFetch("/alerts", 30000);
const { data: timeline } = useFetch("/timeline?hours=48", 60000);
const refresh = () => { r1(); r2(); };
return (
<div style={{ minHeight: "100vh", background: "#0D1117", color: "#E0E0E0", fontFamily: "'Noto Sans KR', -apple-system, sans-serif", padding: "0 24px 40px" }}>
<style>{`
@import url('https://fonts.googleapis.com/css2?family=Noto+Sans+KR:wght@300;400;500;700;800;900&family=JetBrains+Mono:wght@400;600;700&display=swap');
* { box-sizing: border-box; margin: 0; }
::-webkit-scrollbar { width: 6px; }
::-webkit-scrollbar-track { background: transparent; }
::-webkit-scrollbar-thumb { background: rgba(255,255,255,0.1); border-radius: 3px; }
body { background: #0D1117 !important; }
`}</style>
<div style={{ maxWidth: 1400, margin: "0 auto" }}>
<Header onRefresh={refresh} />
{/* Tabs */}
<div style={{ display: "flex", gap: 4, marginBottom: 24 }}>
<Tab active={tab === "overview"} label="📊 종합" onClick={() => setTab("overview")} />
<Tab active={tab === "recommend"} label="🏆 추천종목" onClick={() => setTab("recommend")} />
<Tab active={tab === "news"} label="📰 뉴스피드" onClick={() => setTab("news")} />
<Tab active={tab === "alerts"} label="🚨 알림" onClick={() => setTab("alerts")} />
</div>
{/* Overview */}
{tab === "overview" && (
<div style={{ display: "flex", flexDirection: "column", gap: 24 }}>
{/* Summary Cards */}
<div style={{ display: "grid", gridTemplateColumns: "repeat(auto-fit, minmax(180px, 1fr))", gap: 12 }}>
<SummaryCard icon="📰" label="분석 뉴스" value={summary?.total || 0} sub="최근 7일" />
<SummaryCard icon="▲" label="호재" value={summary?.positive || 0} color="#00E676" sub={`${summary?.sentiment_ratio || 50}%`} />
<SummaryCard icon="▼" label="악재" value={summary?.negative || 0} color="#FF5252" />
<SummaryCard icon="📋" label="DART 공시" value={summary?.dart || 0} />
<SummaryCard icon="🏢" label="분석 종목" value={summary?.stocks_analyzed || 0} />
</div>
{/* Two column: Ranking + Recent */}
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: 16 }}>
{/* Ranking */}
<div style={{ background: "rgba(255,255,255,0.02)", border: "1px solid rgba(255,255,255,0.05)", borderRadius: 16, padding: 20 }}>
<h3 style={{ fontSize: 14, fontWeight: 700, color: "#90A4AE", marginBottom: 16 }}>🏆 종목 랭킹 (최신)</h3>
{ranking && ranking.length > 0 ? ranking.slice(0, 10).map((item, i) => (
<div key={i} style={{ display: "flex", alignItems: "center", gap: 12, padding: "10px 0", borderBottom: "1px solid rgba(255,255,255,0.03)" }}>
<div style={{ width: 24, height: 24, borderRadius: "50%", background: i < 3 ? "rgba(0,230,118,0.15)" : "rgba(255,255,255,0.04)", display: "flex", alignItems: "center", justifyContent: "center", fontSize: 11, fontWeight: 700, color: i < 3 ? "#00E676" : "#546E7A" }}>{i + 1}</div>
<div style={{ flex: 1 }}>
<div style={{ fontSize: 13, fontWeight: 600, color: "#CFD8DC" }}>{item.stock_name}</div>
<div style={{ fontSize: 10, color: "#546E7A" }}>{item.stock_code}</div>
</div>
<div style={{ textAlign: "right" }}>
<div style={{ fontSize: 14, fontWeight: 700, color: (REC_MAP[item.recommendation] || {}).color || "#78909C" }}>{item.total_score?.toFixed(1)}</div>
<div style={{ fontSize: 10, color: (REC_MAP[item.recommendation] || {}).color || "#78909C" }}>{item.recommendation}</div>
</div>
</div>
)) : <div style={{ color: "#546E7A", fontSize: 13, padding: 20, textAlign: "center" }}>데이터 수집 ...</div>}
</div>
{/* Recent News */}
<div style={{ background: "rgba(255,255,255,0.02)", border: "1px solid rgba(255,255,255,0.05)", borderRadius: 16, padding: 20, maxHeight: 500, overflow: "auto" }}>
<h3 style={{ fontSize: 14, fontWeight: 700, color: "#90A4AE", marginBottom: 16 }}>📰 최근 분석</h3>
{recent && recent.length > 0 ? recent.slice(0, 15).map((item, i) => (
<NewsItem key={i} item={item} />
)) : <div style={{ color: "#546E7A", fontSize: 13, padding: 20, textAlign: "center" }}>데이터 수집 ...</div>}
</div>
</div>
</div>
)}
{/* Recommendations */}
{tab === "recommend" && (
<div>
<div style={{ fontSize: 12, color: "#546E7A", marginBottom: 16 }}>뉴스 감성 + DART 공시 + 가격 모멘텀 기반 종합 점수</div>
<div style={{ display: "grid", gridTemplateColumns: "repeat(auto-fill, minmax(320px, 1fr))", gap: 12 }}>
{recs && recs.length > 0 ? recs.map((item, i) => (
<RecommendationCard key={i} item={item} />
)) : <div style={{ color: "#546E7A", fontSize: 13, padding: 40, textAlign: "center", gridColumn: "1/-1" }}>추천 데이터가 아직 없습니다. 16:30 자동 산출 업데이트됩니다.</div>}
</div>
</div>
)}
{/* News Feed */}
{tab === "news" && (
<div style={{ background: "rgba(255,255,255,0.02)", border: "1px solid rgba(255,255,255,0.05)", borderRadius: 16, padding: 20 }}>
<h3 style={{ fontSize: 14, fontWeight: 700, color: "#90A4AE", marginBottom: 16 }}>전체 뉴스 피드</h3>
{recent && recent.length > 0 ? recent.map((item, i) => (
<NewsItem key={i} item={item} />
)) : <div style={{ color: "#546E7A", fontSize: 13, padding: 40, textAlign: "center" }}>뉴스 데이터 수집 ...</div>}
</div>
)}
{/* Alerts */}
{tab === "alerts" && (
<div>
<div style={{ fontSize: 12, color: "#546E7A", marginBottom: 16 }}>강도 3 이상 뉴스/공시 (최근 24시간)</div>
{alerts && alerts.length > 0 ? alerts.map((item, i) => (
<AlertItem key={i} item={item} />
)) : <div style={{ background: "rgba(255,255,255,0.02)", borderRadius: 16, padding: 40, textAlign: "center", color: "#546E7A", fontSize: 13 }}>현재 긴급 알림이 없습니다.</div>}
</div>
)}
{/* Footer */}
<div style={{ marginTop: 40, padding: "16px 0", borderTop: "1px solid rgba(255,255,255,0.04)", textAlign: "center", fontSize: 11, color: "#37474F" }}>
Trading AI System 뉴스/공시 기반 자동 분석 투자 판단의 참고 자료로만 활용하세요
</div>
</div>
</div>
);
}
+8
View File
@@ -0,0 +1,8 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir --default-timeout=180 --retries=5 -r requirements.txt
COPY . .
EXPOSE 8383
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8383", "--workers", "1", "--log-level", "info"]
+838
View File
@@ -0,0 +1,838 @@
"""
US Market Sync Service (port 8383, 172.30.0.24)
미국증시 한국증시 동조 시그널 생성:
A) 섹터 ETF 동조 (SOXX, XBI, LIT 14) 한국 같은 섹터에 ±5
B) 개별 페어 (NVDASK하이닉스 ) 60 회귀 베타 ±10
D) 자동 페어 발굴 (코스피200 × S&P500 60 상관계수)
매일 KST 07:30 미국 정규장 마감 수집, 08:00 시그널 계산.
"""
import os
import asyncio
import json
from datetime import date, datetime, timedelta
from typing import Optional, List, Dict
import asyncpg
import orjson
import structlog
from fastapi import FastAPI, Query, BackgroundTasks
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.cron import CronTrigger
from pytz import timezone
import httpx
import pandas as pd
import numpy as np
from scipy import stats
# ─────────────────────────────────────────────────────────────
# 설정
# ─────────────────────────────────────────────────────────────
PG = {
"host": os.getenv("POSTGRES_HOST", "postgres"),
"port": int(os.getenv("POSTGRES_PORT", 5432)),
"database": os.getenv("POSTGRES_DB", "trading_ai"),
"user": os.getenv("POSTGRES_USER", "kyu"),
"password": os.getenv("POSTGRES_PASSWORD", ""),
}
KST = timezone("Asia/Seoul")
FINNHUB_KEY = os.getenv("FINNHUB_API_KEY", "")
FINNHUB_BASE = "https://finnhub.io/api/v1"
AV_KEY = os.getenv("ALPHAVANTAGE_API_KEY", "")
AV_BASE = "https://www.alphavantage.co/query"
AV_DAILY_LIMIT = 25 # free tier: 25 calls/day
logger = structlog.get_logger()
app = FastAPI(title="US Market Sync")
pg_pool: Optional[asyncpg.Pool] = None
scheduler = AsyncIOScheduler(timezone=KST)
# ─────────────────────────────────────────────────────────────
# ETF → 한국 섹터 키워드 매핑 (dart_corps.sector LIKE 매칭용)
# 섹터 컬럼은 KSIC 한글 (예: "반도체 및 전자집적회로 제조업")
# ─────────────────────────────────────────────────────────────
SECTOR_ETFS: Dict[str, Dict] = {
"SOXX": {"keywords": ["반도체", "전자집적", "전자부품"],
"desc": "iShares Semiconductor"},
"SMH": {"keywords": ["반도체", "전자집적"],
"desc": "VanEck Semiconductor"},
"XLK": {"keywords": ["소프트웨어", "정보서비스", "컴퓨터"],
"desc": "Tech Select Sector"},
"QQQ": {"keywords": ["소프트웨어", "인터넷", "포털"],
"desc": "Nasdaq 100"},
"XBI": {"keywords": ["바이오", "생물의약", "의약품"],
"desc": "S&P Biotech"},
"IBB": {"keywords": ["바이오", "생물의약"],
"desc": "Nasdaq Biotech"},
"LIT": {"keywords": ["전지", "축전지", "이차전지"],
"desc": "Global Lithium"},
"XLE": {"keywords": ["석유", "정제", "가스", "원유"],
"desc": "Energy Select"},
"XLF": {"keywords": ["은행", "보험", "증권", "금융"],
"desc": "Financial Select"},
"XLV": {"keywords": ["의료", "병원", "의료용기기"],
"desc": "Health Care Select"},
"XLI": {"keywords": ["기계", "산업용", "건설"],
"desc": "Industrials Select"},
"XLP": {"keywords": ["식품", "음료", "가공식품"],
"desc": "Consumer Staples"},
"XLY": {"keywords": ["자동차", "여가", "의류", "백화점"],
"desc": "Consumer Discretionary"},
"ITA": {"keywords": ["항공", "방위", "조선"],
"desc": "Aerospace & Defense"},
}
# ─────────────────────────────────────────────────────────────
# 검증된 핵심 페어 (seed) — 학술/시장 통념상 강한 동조
# (us_ticker, kr_code, kr_name_hint) kr_name_hint는 검증용 코멘트
# ─────────────────────────────────────────────────────────────
SEED_PAIRS: List = [
# 반도체
("NVDA", "000660"), # SK하이닉스
("NVDA", "005930"), # 삼성전자
("AMD", "000660"),
("AMD", "005930"),
("MU", "000660"),
("INTC", "005930"),
("TSM", "000660"),
("TSM", "005930"),
# 2차전지
("TSLA", "373220"), # LG에너지솔루션
("TSLA", "247540"), # 에코프로비엠
("TSLA", "006400"), # 삼성SDI
("ALB", "006400"), # 알버말 (리튬)
("ALB", "373220"),
# 자동차
("F", "005380"), # 현대차
("GM", "000270"), # 기아
("TSLA", "012330"), # 현대모비스
("TM", "005380"), # 도요타→현대차
# 인터넷/IT
("GOOGL","035420"), # NAVER
("META", "035720"), # 카카오
("AAPL", "011070"), # LG이노텍
("AAPL", "009150"), # 삼성전기
# 바이오/제약
("PFE", "068270"), # 셀트리온
("MRK", "207940"), # 삼성바이오로직스
# 조선/방산
("LMT", "079550"), # LIG넥스원
("LMT", "329180"), # HD현대중공업
# 철강/소재
("NUE", "005490"), # POSCO홀딩스
# 화학
("DOW", "051910"), # LG화학
("LYB", "011170"), # 롯데케미칼
# 게임/엔터
("NTES", "036570"), # 엔씨소프트
("NTES", "251270"), # 넷마블
("DIS", "035250"), # 강원랜드 (엔터)
# 유통
("AMZN", "139480"), # 이마트
]
# 자동 발굴용 미국 종목 후보 (S&P500 대표 + 한국 영향 큰 종목)
DISCOVERY_US_TICKERS = [
# 반도체
"NVDA", "AMD", "MU", "INTC", "TSM", "AVGO", "QCOM", "TXN", "AMAT", "LRCX", "KLAC",
# 빅테크
"AAPL", "MSFT", "GOOGL", "AMZN", "META", "TSLA", "NFLX", "ORCL", "ADBE", "CRM",
# 금융
"JPM", "BAC", "GS", "MS", "WFC", "C", "BLK",
# 에너지/소재
"XOM", "CVX", "COP", "NUE", "FCX",
# 헬스/바이오
"JNJ", "PFE", "MRK", "ABBV", "LLY", "BMY", "GILD",
# 소비재/유통
"WMT", "COST", "HD", "MCD", "NKE", "SBUX", "DIS",
# 산업/방산
"BA", "LMT", "RTX", "GD", "CAT", "DE",
# 자동차
"F", "GM", "TM",
# 2차전지/리튬
"ALB", "RIVN",
# 화학
"DOW", "LYB",
]
# ─────────────────────────────────────────────────────────────
# DB 초기화
# ─────────────────────────────────────────────────────────────
DDL = """
CREATE TABLE IF NOT EXISTS us_market_daily (
ticker VARCHAR(20),
trade_date DATE,
open_price DOUBLE PRECISION,
close_price DOUBLE PRECISION,
prev_close DOUBLE PRECISION,
change_pct DOUBLE PRECISION,
volume BIGINT,
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (ticker, trade_date)
);
CREATE INDEX IF NOT EXISTS idx_us_daily_ticker ON us_market_daily(ticker, trade_date DESC);
CREATE TABLE IF NOT EXISTS us_sector_etf_map (
etf_ticker VARCHAR(20) PRIMARY KEY,
sector_keywords TEXT[],
description TEXT,
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS us_kr_pairs (
us_ticker VARCHAR(20),
kr_code VARCHAR(10),
beta_60d DOUBLE PRECISION,
correlation_60d DOUBLE PRECISION,
sample_size INTEGER,
source VARCHAR(20) DEFAULT 'seed',
updated_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (us_ticker, kr_code)
);
CREATE INDEX IF NOT EXISTS idx_pairs_kr ON us_kr_pairs(kr_code);
CREATE TABLE IF NOT EXISTS us_overnight_signal (
kr_code VARCHAR(10),
signal_date DATE,
sector_adj DOUBLE PRECISION DEFAULT 0,
pair_adj DOUBLE PRECISION DEFAULT 0,
total_adj DOUBLE PRECISION DEFAULT 0,
contributing_pairs JSONB,
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (kr_code, signal_date)
);
CREATE INDEX IF NOT EXISTS idx_overnight_date ON us_overnight_signal(signal_date DESC);
"""
# ─────────────────────────────────────────────────────────────
# 시작/종료
# ─────────────────────────────────────────────────────────────
@app.on_event("startup")
async def on_start():
global pg_pool
pg_pool = await asyncpg.create_pool(**PG, min_size=2, max_size=10)
async with pg_pool.acquire() as conn:
await conn.execute(DDL)
await seed_etfs_and_pairs()
scheduler.add_job(
collect_us_daily, CronTrigger(hour=7, minute=30),
id="us_collect", replace_existing=True)
scheduler.add_job(
calc_overnight_signals_all, CronTrigger(hour=8, minute=0),
id="overnight_calc", replace_existing=True)
scheduler.add_job(
recalc_pair_betas, CronTrigger(day_of_week="sun", hour=2, minute=0),
id="pair_beta", replace_existing=True)
scheduler.add_job(
discover_new_pairs, CronTrigger(day_of_week="sun", hour=3, minute=0),
id="pair_discover", replace_existing=True)
# 매일 06:00 Alpha Vantage 백필 (25 ticker/일, 60일+ 채워질 때까지)
scheduler.add_job(
backfill_yfinance, CronTrigger(hour=6, minute=0),
id="av_backfill", replace_existing=True)
scheduler.start()
logger.info("us-market.started")
@app.on_event("shutdown")
async def on_stop():
if scheduler.running:
scheduler.shutdown()
if pg_pool:
await pg_pool.close()
# ─────────────────────────────────────────────────────────────
# Seed: ETF + 핵심 페어 등록
# ─────────────────────────────────────────────────────────────
async def seed_etfs_and_pairs():
async with pg_pool.acquire() as conn:
for etf, meta in SECTOR_ETFS.items():
await conn.execute("""
INSERT INTO us_sector_etf_map (etf_ticker, sector_keywords, description)
VALUES ($1, $2, $3)
ON CONFLICT (etf_ticker) DO UPDATE
SET sector_keywords=$2, description=$3, updated_at=NOW()
""", etf, meta["keywords"], meta["desc"])
for us, kr in SEED_PAIRS:
await conn.execute("""
INSERT INTO us_kr_pairs (us_ticker, kr_code, source)
VALUES ($1, $2, 'seed')
ON CONFLICT (us_ticker, kr_code) DO NOTHING
""", us, kr)
logger.info("seed.done", etfs=len(SECTOR_ETFS), pairs=len(SEED_PAIRS))
# ─────────────────────────────────────────────────────────────
# Finnhub 헬퍼 — /quote (무료 OK, /stock/candle은 2024년부터 유료)
# 응답: c(current), pc(prev_close), dp(percent), o(open), h, l, t
# Free tier: 60 calls/분
# ─────────────────────────────────────────────────────────────
async def fetch_finnhub_quote(client: httpx.AsyncClient, ticker: str
) -> Optional[dict]:
"""Finnhub /quote 호출. 실패 시 None."""
if not FINNHUB_KEY:
return None
try:
r = await client.get(f"{FINNHUB_BASE}/quote", params={
"symbol": ticker, "token": FINNHUB_KEY,
}, timeout=15)
except Exception as e:
logger.warning("finnhub.req_err", ticker=ticker, err=str(e))
return None
if r.status_code == 429:
await asyncio.sleep(5)
return None
if r.status_code != 200:
return None
try:
j = r.json()
except Exception:
return None
# 휴장/잘못된 ticker 시 c=0
if not j or j.get("c", 0) <= 0:
return None
return j
# ─────────────────────────────────────────────────────────────
# 수집: /quote으로 미국 일간 종가 누적
# 매일 호출하면 us_market_daily에 시계열 점진 누적 → 베타 학습 가능
# ─────────────────────────────────────────────────────────────
async def collect_us_daily(days: int = 1):
"""매일 KST 07:30 호출. days 인자는 호환용(무시) — /quote은 단일 시점."""
if not FINNHUB_KEY:
logger.error("us.no_api_key")
return {"saved": 0, "err": "FINNHUB_API_KEY missing — set in .env"}
tickers = sorted(
set(SECTOR_ETFS.keys())
| {t for t, _ in SEED_PAIRS}
| set(DISCOVERY_US_TICKERS)
)
saved = 0
failed: List[str] = []
async with httpx.AsyncClient() as client:
for i, ticker in enumerate(tickers):
q = await fetch_finnhub_quote(client, ticker)
if not q:
failed.append(ticker)
else:
# t 타임스탬프(미국 ET 종가 시점) → trade_date
ts = q.get("t", 0)
trade_dt = (datetime.fromtimestamp(ts).date() if ts
else date.today() - timedelta(days=1))
async with pg_pool.acquire() as conn:
await conn.execute("""
INSERT INTO us_market_daily
(ticker, trade_date, open_price, close_price,
prev_close, change_pct, volume)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (ticker, trade_date) DO UPDATE
SET open_price=$3, close_price=$4, prev_close=$5,
change_pct=$6
""", ticker, trade_dt,
float(q.get("o", 0) or 0), float(q["c"]),
float(q.get("pc", 0) or 0), float(q.get("dp", 0) or 0),
0)
saved += 1
# Rate limit: 60/분 → 1.1초 간격
if i < len(tickers) - 1:
await asyncio.sleep(1.1)
logger.info("us.collected", rows=saved, ok=len(tickers) - len(failed),
failed=len(failed))
return {"saved": saved, "tickers": len(tickers),
"ok": len(tickers) - len(failed), "failed": failed[:10]}
# ─────────────────────────────────────────────────────────────
# Alpha Vantage 백필 — TIME_SERIES_DAILY로 ticker당 100일 히스토리.
# Free tier 25 calls/day → 자동으로 매일 25개씩 분할 처리 (75 ticker × 3일).
# ─────────────────────────────────────────────────────────────
async def fetch_av_daily(client: httpx.AsyncClient, ticker: str
) -> Optional[List[dict]]:
"""Alpha Vantage TIME_SERIES_DAILY 호출. 실패/한도초과 시 None."""
try:
r = await client.get(AV_BASE, params={
"function": "TIME_SERIES_DAILY",
"symbol": ticker,
"outputsize": "compact", # 100일치 (full=20+년)
"apikey": AV_KEY,
}, timeout=20)
except Exception as e:
logger.warning("av.req_err", ticker=ticker, err=str(e))
return None
if r.status_code != 200:
return None
try:
j = r.json()
except Exception:
return None
if "Note" in j or "Information" in j or "Error Message" in j:
logger.warning("av.limit_or_err",
ticker=ticker, msg=str(j)[:200])
return None
ts = j.get("Time Series (Daily)")
if not ts:
return None
rows = []
for dt_str, ohlcv in ts.items():
try:
rows.append({
"trade_date": date.fromisoformat(dt_str),
"open": float(ohlcv["1. open"]),
"high": float(ohlcv["2. high"]),
"low": float(ohlcv["3. low"]),
"close": float(ohlcv["4. close"]),
"volume": int(float(ohlcv["5. volume"])),
})
except (KeyError, ValueError):
continue
rows.sort(key=lambda x: x["trade_date"])
return rows
async def backfill_yfinance(days: int = 180, max_tickers: int = 0):
"""Alpha Vantage로 히스토리 백필.
- max_tickers=0 (기본): AV_DAILY_LIMIT(=25)개만 처리 일일 한도 자동 준수
- max_tickers>0: 명시값 사용
- 이미 60+ 데이터 있는 ticker는 건너뜀 3일치 분산 자동 진행
"""
if not AV_KEY:
return {"saved": 0, "err": "ALPHAVANTAGE_API_KEY missing — set in .env"}
all_tickers = sorted(
set(SECTOR_ETFS.keys())
| {t for t, _ in SEED_PAIRS}
| set(DISCOVERY_US_TICKERS)
)
# 이미 60일+ 누적된 ticker는 스킵
async with pg_pool.acquire() as conn:
rows = await conn.fetch(
"SELECT ticker, COUNT(*) AS n FROM us_market_daily "
"GROUP BY ticker HAVING COUNT(*) >= 60")
done = {r["ticker"] for r in rows}
pending = [t for t in all_tickers if t not in done]
limit = max_tickers if max_tickers > 0 else AV_DAILY_LIMIT
targets = pending[:limit]
saved = 0
failed: List[str] = []
async with httpx.AsyncClient() as client:
for i, ticker in enumerate(targets):
rows = await fetch_av_daily(client, ticker)
if not rows:
failed.append(ticker)
# 한도 초과면 즉시 중단
if i > 0 and len(failed) > 3 and len(failed) > i // 2:
logger.warning("av.likely_quota", processed=i)
break
else:
prev_close = None
async with pg_pool.acquire() as conn:
async with conn.transaction():
for row in rows:
pc = prev_close if prev_close is not None else 0.0
dp = ((row["close"] - pc) / pc * 100.0) if pc > 0 else 0.0
await conn.execute("""
INSERT INTO us_market_daily
(ticker, trade_date, open_price, close_price,
prev_close, change_pct, volume)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (ticker, trade_date) DO UPDATE
SET open_price=$3, close_price=$4, prev_close=$5,
change_pct=$6, volume=$7
""", ticker, row["trade_date"], row["open"],
row["close"], pc, dp, row["volume"])
saved += 1
prev_close = row["close"]
# AV free tier 5 calls/min → 12초 간격
if i < len(targets) - 1:
await asyncio.sleep(12.5)
logger.info("us.av_backfill", saved=saved,
processed=len(targets) - len(failed),
failed=len(failed),
pending_remaining=len(pending) - len(targets) + len(failed))
return {"saved": saved, "processed": len(targets),
"ok": len(targets) - len(failed),
"failed": failed[:10],
"pending_after": max(0, len(pending) - len(targets) + len(failed)),
"source": "alphavantage"}
# ─────────────────────────────────────────────────────────────
# 시그널 계산: 한국 종목별 overnight 보정 점수
# ─────────────────────────────────────────────────────────────
async def calc_overnight_signals_all(target_date: Optional[date] = None):
"""오늘 자(=어제 미국장 마감) 보정 점수 계산.
A. 섹터 ETF 동조: ETF change_pct 같은 sector 한국 종목에 ±5
B. 페어 동조: 페어별 미국주 change_pct × beta ±10 (집계)
"""
target = target_date or date.today()
async with pg_pool.acquire() as conn:
# 1) 최신 미국 거래일의 ETF/주식 change_pct
us_rows = await conn.fetch("""
SELECT DISTINCT ON (ticker)
ticker, trade_date, change_pct
FROM us_market_daily
WHERE trade_date <= $1 AND change_pct IS NOT NULL
ORDER BY ticker, trade_date DESC
""", target)
us_chg = {r["ticker"]: float(r["change_pct"]) for r in us_rows}
if not us_chg:
logger.warning("overnight.no_us_data")
return {"saved": 0, "err": "no us data"}
# 2) 활성 한국 종목 + 섹터
kr_rows = await conn.fetch("""
SELECT stock_code, sector FROM dart_corps WHERE is_active=true
""")
# 3) 페어 매핑 (us_ticker → list of (kr_code, beta))
pair_rows = await conn.fetch("""
SELECT us_ticker, kr_code, beta_60d, correlation_60d
FROM us_kr_pairs
""")
pairs_by_kr: Dict[str, List] = {}
for r in pair_rows:
kr = r["kr_code"]
pairs_by_kr.setdefault(kr, []).append({
"us": r["us_ticker"],
"beta": float(r["beta_60d"]) if r["beta_60d"] else 1.0,
"corr": float(r["correlation_60d"]) if r["correlation_60d"] else 0.0,
})
# 4) ETF 매핑
etf_rows = await conn.fetch("SELECT etf_ticker, sector_keywords FROM us_sector_etf_map")
saved = 0
for kr in kr_rows:
code = kr["stock_code"]
sector = (kr["sector"] or "")
# A. 섹터 ETF 동조
sector_adj = 0.0
matched_etfs = []
for er in etf_rows:
kws = er["sector_keywords"] or []
if not sector or not kws:
continue
if any(kw and kw in sector for kw in kws):
pct = us_chg.get(er["etf_ticker"])
if pct is None:
continue
matched_etfs.append({"etf": er["etf_ticker"], "pct": pct})
if matched_etfs:
# 매칭된 ETF 평균 변동률 → ±5 클램프
avg_pct = sum(m["pct"] for m in matched_etfs) / len(matched_etfs)
sector_adj = max(-5.0, min(5.0, avg_pct * 1.5))
# B. 페어 베타 기반
pair_adj = 0.0
contributing = []
kr_pairs = pairs_by_kr.get(code, [])
for p in kr_pairs:
pct = us_chg.get(p["us"])
if pct is None:
continue
# 예상 갭 = 미국주 변동률 × 베타
exp_gap = pct * p["beta"]
# 상관계수 가중 (|corr|이 낮으면 신뢰도 ↓)
weight = max(0.3, abs(p["corr"])) if p["corr"] else 0.5
contrib = exp_gap * weight
pair_adj += contrib
contributing.append({
"us": p["us"], "pct": round(pct, 2),
"beta": round(p["beta"], 2),
"corr": round(p["corr"], 2),
"contribution": round(contrib, 2),
})
if contributing:
# 다중 페어 평균 + 클램프
pair_adj = max(-10.0, min(10.0, pair_adj / len(contributing) * 2.0))
total_adj = sector_adj + pair_adj
if abs(total_adj) < 0.1 and not matched_etfs and not contributing:
continue # 영향 없는 종목은 저장 스킵
await conn.execute("""
INSERT INTO us_overnight_signal
(kr_code, signal_date, sector_adj, pair_adj, total_adj, contributing_pairs)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (kr_code, signal_date) DO UPDATE
SET sector_adj=$3, pair_adj=$4, total_adj=$5,
contributing_pairs=$6, created_at=NOW()
""", code, target, sector_adj, pair_adj, total_adj,
json.dumps({"etfs": matched_etfs, "pairs": contributing}))
saved += 1
logger.info("overnight.calculated", saved=saved, date=str(target))
return {"saved": saved, "date": str(target)}
# ─────────────────────────────────────────────────────────────
# 페어 베타 재계산 (주 1회 일요일)
# ─────────────────────────────────────────────────────────────
async def recalc_pair_betas(window_days: int = 60):
"""등록된 페어에 대해 최근 N일 일간수익률로 회귀 → beta, correlation 갱신."""
since = date.today() - timedelta(days=window_days * 2) # 거래일 여유
async with pg_pool.acquire() as conn:
pairs = await conn.fetch("SELECT us_ticker, kr_code FROM us_kr_pairs")
updated, skipped = 0, 0
for p in pairs:
us_t, kr_c = p["us_ticker"], p["kr_code"]
# 미국 시계열
us_rows = await conn.fetch("""
SELECT trade_date, close_price FROM us_market_daily
WHERE ticker=$1 AND trade_date >= $2 ORDER BY trade_date
""", us_t, since)
if len(us_rows) < 30:
skipped += 1
continue
# 한국 시계열 (stock_prices.collected_at)
kr_rows = await conn.fetch("""
SELECT collected_at::date AS dt,
AVG(price)::float AS close
FROM stock_prices
WHERE stock_code=$1 AND collected_at::date >= $2
GROUP BY collected_at::date ORDER BY dt
""", kr_c, since)
if len(kr_rows) < 30:
skipped += 1
continue
us_df = pd.DataFrame([(r["trade_date"], r["close_price"]) for r in us_rows],
columns=["dt", "us"])
kr_df = pd.DataFrame([(r["dt"], r["close"]) for r in kr_rows],
columns=["dt", "kr"])
us_df["dt"] = pd.to_datetime(us_df["dt"])
kr_df["dt"] = pd.to_datetime(kr_df["dt"])
# 한국 종가는 미국 다음날에 영향 받음 → 미국 시계열을 +1일 시프트해서 매칭
us_df["dt"] = us_df["dt"] + pd.Timedelta(days=1)
merged = pd.merge(us_df, kr_df, on="dt", how="inner")
if len(merged) < window_days // 2:
skipped += 1
continue
us_ret = merged["us"].pct_change().dropna()
kr_ret = merged["kr"].pct_change().dropna()
n = min(len(us_ret), len(kr_ret))
if n < 20:
skipped += 1
continue
us_ret, kr_ret = us_ret.iloc[-n:].values, kr_ret.iloc[-n:].values
# 회귀: kr_ret = beta * us_ret + intercept
slope, _, r_val, _, _ = stats.linregress(us_ret, kr_ret)
beta = float(slope)
corr = float(r_val)
await conn.execute("""
UPDATE us_kr_pairs
SET beta_60d=$1, correlation_60d=$2, sample_size=$3, updated_at=NOW()
WHERE us_ticker=$4 AND kr_code=$5
""", beta, corr, n, us_t, kr_c)
updated += 1
logger.info("pair_beta.recalc", updated=updated, skipped=skipped)
return {"updated": updated, "skipped": skipped}
# ─────────────────────────────────────────────────────────────
# 자동 페어 발굴 (월 1회)
# 코스피200 시총상위 50개 × DISCOVERY_US 후보 → |corr|≥0.5인 것만 등록
# ─────────────────────────────────────────────────────────────
async def discover_new_pairs(min_abs_corr: float = 0.5, top_kr: int = 50,
window_days: int = 60):
since = date.today() - timedelta(days=window_days * 2)
async with pg_pool.acquire() as conn:
# 시총상위 한국 종목
top_rows = await conn.fetch("""
SELECT DISTINCT ON (stock_code) stock_code, market_cap
FROM stock_prices
WHERE collected_at::date >= CURRENT_DATE - INTERVAL '7 days'
AND market_cap > 0
ORDER BY stock_code, collected_at DESC
""")
kr_top = sorted(top_rows, key=lambda r: -(r["market_cap"] or 0))[:top_kr]
added = 0
for kr in kr_top:
kr_c = kr["stock_code"]
kr_rows = await conn.fetch("""
SELECT collected_at::date AS dt, AVG(price)::float AS close
FROM stock_prices
WHERE stock_code=$1 AND collected_at::date >= $2
GROUP BY collected_at::date ORDER BY dt
""", kr_c, since)
if len(kr_rows) < 30:
continue
kr_df = pd.DataFrame([(r["dt"], r["close"]) for r in kr_rows],
columns=["dt", "kr"])
kr_df["dt"] = pd.to_datetime(kr_df["dt"])
for us_t in DISCOVERY_US_TICKERS:
us_rows = await conn.fetch("""
SELECT trade_date, close_price FROM us_market_daily
WHERE ticker=$1 AND trade_date >= $2 ORDER BY trade_date
""", us_t, since)
if len(us_rows) < 30:
continue
us_df = pd.DataFrame([(r["trade_date"], r["close_price"]) for r in us_rows],
columns=["dt", "us"])
us_df["dt"] = pd.to_datetime(us_df["dt"]) + pd.Timedelta(days=1)
merged = pd.merge(us_df, kr_df, on="dt", how="inner")
if len(merged) < window_days // 2:
continue
us_ret = merged["us"].pct_change().dropna()
kr_ret = merged["kr"].pct_change().dropna()
n = min(len(us_ret), len(kr_ret))
if n < 20:
continue
slope, _, r_val, _, _ = stats.linregress(
us_ret.iloc[-n:].values, kr_ret.iloc[-n:].values)
if abs(r_val) < min_abs_corr:
continue
await conn.execute("""
INSERT INTO us_kr_pairs
(us_ticker, kr_code, beta_60d, correlation_60d, sample_size, source)
VALUES ($1, $2, $3, $4, $5, 'discovered')
ON CONFLICT (us_ticker, kr_code) DO UPDATE
SET beta_60d=$3, correlation_60d=$4, sample_size=$5, updated_at=NOW()
""", us_t, kr_c, float(slope), float(r_val), n)
added += 1
logger.info("pair_discover.done", added=added)
return {"added": added}
# ─────────────────────────────────────────────────────────────
# REST API
# ─────────────────────────────────────────────────────────────
@app.get("/health")
async def health():
return {"ok": True, "service": "us-market", "ts": datetime.now(KST).isoformat()}
@app.post("/collect")
async def manual_collect(days: int = Query(default=7, ge=1, le=365),
bg: BackgroundTasks = None):
"""수동 수집. days=7 일상, 백필은 180 권장."""
if bg:
bg.add_task(collect_us_daily, days)
return {"status": "queued", "days": days}
return await collect_us_daily(days)
@app.post("/collect/backfill")
async def backfill(days: int = Query(default=180, ge=30, le=365),
bg: BackgroundTasks = None):
"""대규모 백필 — 페어 베타 학습용 180일 권장. 76 ticker × 1.1초 ≈ 85초."""
if bg:
bg.add_task(collect_us_daily, days)
return {"status": "queued", "days": days}
return await collect_us_daily(days)
@app.post("/collect/yfinance-backfill")
async def yfinance_backfill_ep(days: int = Query(default=180, ge=30, le=730),
bg: BackgroundTasks = None):
"""yfinance로 일괄 히스토리 다운로드 (페어 베타 학습 초기 시드용)."""
if bg:
bg.add_task(backfill_yfinance, days)
return {"status": "queued", "days": days}
return await backfill_yfinance(days)
@app.post("/signal/calculate")
async def manual_signal(target: Optional[str] = None):
d = date.fromisoformat(target) if target else date.today()
return await calc_overnight_signals_all(d)
@app.get("/signal/{kr_code}")
async def get_signal(kr_code: str, days: int = Query(default=7, ge=1, le=90)):
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT signal_date, sector_adj, pair_adj, total_adj, contributing_pairs
FROM us_overnight_signal
WHERE kr_code=$1 AND signal_date >= CURRENT_DATE - $2::int
ORDER BY signal_date DESC
""", kr_code, days)
return [dict(r) for r in rows]
@app.get("/signal/latest")
async def latest_signals(limit: int = Query(default=50, le=500)):
async with pg_pool.acquire() as conn:
rows = await conn.fetch("""
SELECT DISTINCT ON (kr_code)
kr_code, signal_date, sector_adj, pair_adj, total_adj
FROM us_overnight_signal
ORDER BY kr_code, signal_date DESC
LIMIT $1
""", limit)
return [dict(r) for r in rows]
@app.get("/pairs")
async def list_pairs(kr_code: Optional[str] = None, us_ticker: Optional[str] = None):
async with pg_pool.acquire() as conn:
if kr_code:
rows = await conn.fetch(
"SELECT * FROM us_kr_pairs WHERE kr_code=$1 ORDER BY ABS(correlation_60d) DESC NULLS LAST",
kr_code)
elif us_ticker:
rows = await conn.fetch(
"SELECT * FROM us_kr_pairs WHERE us_ticker=$1 ORDER BY ABS(correlation_60d) DESC NULLS LAST",
us_ticker)
else:
rows = await conn.fetch(
"SELECT * FROM us_kr_pairs ORDER BY ABS(correlation_60d) DESC NULLS LAST LIMIT 200")
return [dict(r) for r in rows]
@app.post("/pairs/recalc-beta")
async def manual_recalc(window: int = Query(default=60, ge=20, le=250)):
return await recalc_pair_betas(window)
@app.post("/pairs/discover")
async def manual_discover(min_corr: float = Query(default=0.5, ge=0.1, le=0.9),
top_kr: int = Query(default=50, ge=10, le=200)):
return await discover_new_pairs(min_corr, top_kr)
@app.get("/etfs")
async def list_etfs():
async with pg_pool.acquire() as conn:
rows = await conn.fetch("SELECT * FROM us_sector_etf_map ORDER BY etf_ticker")
return [dict(r) for r in rows]
@app.get("/etfs/{etf}/latest")
async def etf_latest(etf: str):
async with pg_pool.acquire() as conn:
row = await conn.fetchrow("""
SELECT * FROM us_market_daily WHERE ticker=$1
ORDER BY trade_date DESC LIMIT 1
""", etf.upper())
return dict(row) if row else {"err": "no data"}
@app.get("/stats")
async def stats_endpoint():
async with pg_pool.acquire() as conn:
r1 = await conn.fetchrow(
"SELECT COUNT(*) AS rows, COUNT(DISTINCT ticker) AS tickers,"
" MIN(trade_date) AS earliest, MAX(trade_date) AS latest FROM us_market_daily")
r2 = await conn.fetchrow(
"SELECT COUNT(*) AS total, COUNT(*) FILTER (WHERE source='seed') AS seed,"
" COUNT(*) FILTER (WHERE source='discovered') AS discovered,"
" COUNT(*) FILTER (WHERE beta_60d IS NOT NULL) AS with_beta"
" FROM us_kr_pairs")
r3 = await conn.fetchrow(
"SELECT COUNT(*) AS rows, COUNT(DISTINCT kr_code) AS codes,"
" MAX(signal_date) AS latest FROM us_overnight_signal")
return {"us_daily": dict(r1), "pairs": dict(r2), "signals": dict(r3)}
+11
View File
@@ -0,0 +1,11 @@
fastapi==0.111.0
uvicorn[standard]==0.30.1
asyncpg==0.29.0
apscheduler==3.10.4
structlog==24.2.0
orjson==3.10.3
httpx==0.27.0
pandas==2.2.2
numpy==1.26.4
scipy==1.13.1
pytz==2024.1