martin/Sharepoint-Download-Tool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Opdater dokumentation (README og GEMINI.md) med Production Ready specifikationer

Browse Source
This commit is contained in:
Martin Tranberg
2026-03-26 15:44:30 +01:00
parent 1ed21e4184
commit 4c52b0c8db
2 changed files with 33 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
GEMINI.md
View File

@@ -1,51 +1,36 @@
# SharePoint Download Tool
# SharePoint Download Tool - Technical Documentation
A Python-based utility designed to recursively download folders and files from a specific SharePoint Online Site using the Microsoft Graph API.
A production-ready Python utility for robust synchronization of SharePoint Online folders using Microsoft Graph API.
## Project Overview
* **Purpose:** Automates the synchronization of specific SharePoint document library folders to a local directory.
* **Purpose:** Enterprise-grade synchronization tool for local mirroring of SharePoint content.
* **Technologies:**
* **Python 3.x**
* **Microsoft Graph API:** Used for robust data access.
* **MSAL (Microsoft Authentication Library):** Handles Entra ID (Azure AD) authentication using Client Credentials flow.
* **Requests:** Manages HTTP streaming for large file downloads.
* **Architecture:**
* `download_sharepoint.py`: The core script that orchestrates authentication, site/drive discovery, and recursive folder traversal.
* `connection_info.txt`: Centralized configuration file for credentials and target paths.
* `requirements.txt`: Defines necessary Python dependencies.
* **Microsoft Graph API:** Advanced REST API for SharePoint data.
* **MSAL:** Secure authentication using Azure AD Client Credentials.
* **Requests:** High-performance HTTP client with streaming and Range header support.
* **ThreadPoolExecutor:** Parallel file processing for optimized throughput.
## Core Features (Production Ready)
1. **Resumable Downloads:** Implements HTTP `Range` headers to resume partially downloaded files, critical for multi-gigabyte assets.
2. **Reliability:** Includes a custom `retry_request` decorator for Exponential Backoff, handling throttling (429) and transient network errors.
3. **Concurrency:** Multi-threaded architecture (5 workers) for simultaneous scanning and downloading.
4. **Pagination:** Full support for OData pagination, ensuring complete folder traversal regardless of item count.
5. **Logging & Audit:** Integrated Python `logging` to `sharepoint_download.log` and structured CSV reports for error auditing.
## Building and Running
### Prerequisites
* Python 3.x installed.
* A registered application in Microsoft Entra ID with `Sites.Read.All` (or higher) application permissions.
### Setup
1. **Install Dependencies:**
```bash
pip install -r requirements.txt
```
2. **Configure Connection:**
Edit `connection_info.txt` with your specific details:
* `TENANT_ID`, `CLIENT_ID`, `CLIENT_SECRET`
* `SITE_URL`: Full URL to the SharePoint site.
* `DOCUMENT_LIBRARY`: The name of the target library (e.g., "Documents").
* `FOLDERS_TO_DOWNLOAD`: Comma-separated list of folder names to sync.
* `LOCAL_PATH`: The destination path on your local machine.
1. **Dependencies:** `pip install -r requirements.txt`
2. **Configuration:** Use `connection_info.template.txt` to create `connection_info.txt`.
### Execution
Run the main download script:
```bash
python download_sharepoint.py
```
### Validation
After execution, a CSV report named `download_report_YYYYMMDD_HHMMSS.csv` is generated, detailing any failed downloads or size mismatches for verification.
`python download_sharepoint.py`
## Development Conventions
* **Authentication:** Always use the Graph API with MSAL for app-only authentication.
* **Error Handling:** All file and folder operations should be wrapped in try-except blocks, with errors logged to the generated CSV report.
* **Verification:** Post-download verification is performed by comparing the local file size against the `size` property returned by the Graph API.
* **Security:** Never commit `connection_info.txt` or any file containing secrets. Use the provided `.gitignore`.
* **Error Handling:** Always use the `safe_get` (retry-wrapped) method for Graph API calls.
* **Thread Safety:** Use `report_lock` when updating the shared error list from worker threads.
* **Logging:** Prefer `logger.info()` or `logger.error()` over `print()` to ensure persistence in `sharepoint_download.log`.
* **Integrity:** Always verify file integrity using `size` and `quickXorHash` where available.

23
README.md
View File

@@ -1,19 +1,16 @@
# SharePoint Folder Download Tool
Dette script gør det muligt at downloade specifikke mapper fra et SharePoint dokumentbibliotek til din lokale computer ved hjælp af Microsoft Graph API. Scriptet understøtter rekursiv download, filvalidering (størrelsestjek) og genererer en fejlrapport, hvis noget går galt.
Dette script gør det muligt at downloade specifikke mapper fra et SharePoint dokumentbibliotek til din lokale computer ved hjælp af Microsoft Graph API. Scriptet er designet til professionelt brug med fokus på hastighed, stabilitet og dataintegritet.
## Funktioner
* **Rekursiv Download:** Downloader alle undermapper og filer i de valgte mapper.
* **Filnavn-sanitering:** Håndterer ulovlige tegn (f.eks. `<`, `>`, `:`, `"`, `/`, `\`, `|`, `?`, `*`) og Unicode-mellemrum, så SharePoint-filer altid kan gemmes på Windows.
* **Long Path Support:** Understøtter filstier på over 260 tegn på Windows ved brug af `\\?\` præfiks.
* **Status i Realtid:** Viser en progress-indikator med antal tjekkede, downloadede, skippede og fejlede filer, samt den aktuelle sti, der arbejdes på.
* **Netværksstabilitet:** Tjekker om destinationsstien er tilgængelig ved opstart og håndterer fejl, hvis f.eks. et netværksdrev mister forbindelsen under kørslen.
* **Smart Skip:** Skipper automatisk filer, der allerede findes lokalt med den korrekte filstørrelse (sparer tid ved genstart).
* **Token Refresh:** Håndterer automatisk fornyelse af adgangstoken, så lange kørsler ikke afbrydes af timeout.
* **Fejlrapportering:** Genererer en CSV-fil med detaljer om eventuelle fejl og specifikke fejlkoder (f.eks. `[Error 22]` eller netværksfejl).
* **Dataintegritet:** Sammenligner lokal filstørrelse med SharePoint-størrelsen for at sikre korrekt overførsel.
* **Entra ID Integration:** Benytter MSAL for sikker godkendelse via Client Credentials flow.
* **Paralleldownload:** Benytter `ThreadPoolExecutor` (default 5 tråde) for markant højere overførselshastighed.
* **Resume Download:** Understøtter HTTP `Range` headers, så afbrudte downloads af store filer (f.eks. >50GB) genoptages fra det sidste byte i stedet for at starte forfra.
* **Exponential Backoff:** Håndterer automatisk Microsoft Graph throttling (`429 Too Many Requests`) og netværksfejl med intelligente genforsøg.
* **Struktureret Logging:** Gemmer detaljerede logs i `sharepoint_download.log` samt en CSV-fejlrapport for hver kørsel.
* **Paginering:** Håndterer automatisk mapper med mere end 200 elementer via `@odata.nextLink`.
* **Smart Skip & Integritet:** Skipper filer der allerede findes lokalt med korrekt størrelse, og forbereder til hash-validering (QuickXorHash).
* **Entra ID Integration:** Benytter MSAL for sikker godkendelse via Client Credentials flow med automatisk token-refresh.
## Installation
@@ -55,7 +52,9 @@ Kør scriptet med:
python download_sharepoint.py
```
Efter kørsel vil en CSV-rapport (f.eks. `download_report_20260326.csv`) være tilgængelig, hvis der er opstået fejl.
### Logfiler
* `sharepoint_download.log`: Teknisk log over alle handlinger og fejl.
* `download_report_YYYYMMDD_HHMMSS.csv`: En hurtig oversigt over filer der fejlede.
## Sikkerhed
Husk at `.gitignore` er sat op til at ignorere `connection_info.txt`, så dine adgangskoder ikke uploades til Git.