AntiPublic
AntiPublic is a module that helps you check the uniqueness of your databases. It can check any database for uniqueness (url:log:pass, log:pass, etc.). The module loads databases into memory and checks them for uniqueness, making it an essential tool for data cleansing.
This module is ideal when you need to ensure your working databases contain only unique entries or to verify new data against existing databases.
Please note that this module can require significant memory resources when processing large databases.
Options
File or directory with new databases that you want to check for uniqueness.
Usage Guide
Follow these steps to effectively use the AntiPublic module:
Select Source Database
Choose the file or directory containing new databases that you want to check for uniqueness.
Select Stored Database
Choose the file or directory containing your main databases for comparison.
Select Mode
Choose between Fast and LowMemory modes based on your system resources and database size.
- For smaller databases or systems with ample RAM, select Fast
- For very large databases or systems with limited RAM, select LowMemory
Configure Special Options
Enable Mail:Pass if working with email:password databases.
Enable Save Private if you want to preserve the original database structure in the output.
Execute and Verify
Run the module and check the output directory for results. The output will contain only the unique entries not found in the stored database.
Example Workflow
When you acquire a new database and want to check if it contains unique entries compared to your existing collection:
- Set the new database as Source
- Set your existing database collection as Stored
- Select Fast mode if your system has sufficient RAM
- Run the module
- The output will contain only entries that are unique to the new database
Performance Tip
For extremely large databases, consider breaking them into smaller chunks using the Splitter module before processing with AntiPublic in LowMemory mode.