I joined the offensive volunteer unit and managed a small team of blackhats to help Ukraine's military conduct digital operations against invading Russian forces.
Also worked with another volunteer unit designing and implementing custom ESP32 and ESP8266 microelectronics systems, for purposes such as obscured communications systems, area monitoring, location tracking, wide-area audio source triangulation, and various drone systems.
Notable accomplishments across both units have been significant but under NDA.
I built and supported custom applications for the Computer Science and Engineering Colleges at Cornell. I also maintained infrastructure, upgraded ancient applications to enhance security and reliability, introduced CI/CD and container solutions, optimized systems and reduced overhead expenses, and helped simplify the entire architecture.
Typical tasks and technologies included OS maintenance and management for a wide variety of Windows servers, as well as Debian, Ubuntu, Redhat, and CentOS servers. Also handled server and service security considerations, performed database administration on MySQL, PostgreSQL, Oracle, and SQL Server instances. Was often given something a teacher or student had made who wanted it to be made publicly, with no consistency in what it was built with and it not being designed with security or public access in mind.
Some notable accomplishments were the migration of hundreds of websites from individual servers to a distributed redundant framework, transitioning software using individual database instances to centralized database servers, instituting regular and increasingly automated OS and software upgrades, transitioning software solutions to using centralized two-factor capable authentication systems, and pushing the department toward containers and orchestration.
Techendeavors was a business cooperative that I started that built turn-key profitable websites, operated them until they were consistently profitable, and then automated their operation or sold them. Unlike standard development studios, we didn't accept commissions for work. We identified potential business opportunities and democratically decided on what to focus our energy on.
Typical operations and technologies included managing a team of half a dozen people, building systems using the predetermined standard technologies. Our typical stack in 2020 was containers for static front-end assets, API endpoints, Redis for session storage, and PostgreSQL or SQLite for database systems. All businesses were built to be portable and be able to be provisioned either on customer-owned hardware or any number of cloud services.
All cooperative members received shares for the effort they put into development that determined dividends returned upon sale. The group build over 250 businesses, of which over 170 were sold to entrepreneurs. 20 were run by the cooperative itself. From incarnation, Techendeavors generated over $10mm in proceeds for it's members.
Advising cPay.click, an advanced Cryptocurrency platform that combines the best of ecommerce and cryptocurrency into one integrated platform.
Primary developer and equity partner of CoinMall.io, the Crypto Marketplace for Digital Goods. Several categories of products were offered (commissioned, digital with automatic delivery, auctions, and regular sales). Built to be similar to Amazon, Ebay, and Shopify.
Built from the start with security in mind and to easily scale, initial users hit a pool of containers with static front-end assets and API endpoints for read-only data. Upon creation of shopping cart, or after logging in, a container was created for each customer to keep their experience isolated. System was engineered to be multi-homed using Anycast so different geographic locations could be served by regional providers.
I was particularly proud of the systems integration between container orchestration, BGP announcements and Anycast system, and strong security isolation techniques. Any security flaws on the application layer would only let the attacker gain access to their isolated container and their own data, not the data of any other users.
Service that sold gift cards for gaming and media platforms in exchange for Bitcoin. Retired when the primary product, Steam Wallet Codes, became obsolete because the Steam platform started accepting Bitcoin directly (for a while).
Everything was built bespoke, with a locally hosted bitcoin daemon that monitored on-chain transactions. Front-end was written in PHP, interfacing with a custom built internal Python API. System was designed to allow people to make orders without creating an account. It would analyze transactions to determine a security score to determine how "risky" a transaction was. Once a certain score threshold was passed, the gift card code was emailed to the recipient.
The system performed admirably for 2 years without ever being compromised by any of the many hundreds of attempts to cheat the system. With an average 7% profit for each transaction, the system was profitable from it's initial concept and over $4mm of transactions were handled.
Contributed to the core code-base of an advanced p2p payments platform and a credit card payment processor. Designed database schema, picked and implemented Laravel framework, designed the encryption methods and procedures, and implemented the basics of that security.
Built the infrastructure to power a new generation of Authentication Providers to allow businesses to utilize Zero Knowledge Identity Verification. Goal was to allow users to prove their identity to an identity provider once and not have to prove it again to any websites that accepted the BlockAuth compatible logins. Benefits to business partners included guarantees that users were real people, unique, and meet any legal requirements to use the service.
Recipient of $250,000 BitAngels.co Angel Investment.
Responsible for implementing methods to keep 500+ client machines and 30+ servers up-to-date and operational, monitoring network and server health, prevention of virus and malware outbreaks, and investigating improvements to procedures and technology using current best practices.
Service that sold game and media oriented gift cards for Bitcoin. System was mostly automated and earned a profit from it's inception.
Directory of Laravel programming resources in the manner of the original Yahoo Directory
Accounting of known Bitcoin thefts and losses due to negligence throughout history.
Service that allowed users to upload documents, images, or software distributions. The uploads were then digitally signed, and proof of their existence and file hashes are published in the Bitcoin Blockchain. Users also could opt to receive a notarized document attesting to the details of their upload. API was developed to allow for full automation
Software Development Languages & Platforms
Language Learning Models (LLM)
Set up a small, but diverse GPU collection for AI/ML work and utilized it with several different systems, such as TensorFlow, PyTorch, Caffe, Pygmalion, and many more. Have build several projects that do image recognition. Built a system that did recognition and categorization of things based on a combination of factors including object self-reported identification, image, sound, heat, and movement. Have done some work on language analysis and generation, including intelligent chatbots, and built a proactive home automation system with conversational capabilities. Also built a distributed sound source triangulation system that used Machine Learning to identify the location and nature of the source of the sound.
A firm believer in the 3-2-1 backup rule, I’m well familiar with many commercial and open-source solutions such as Amanda, Bareos & Bacula, Barman (for PostgreSQL servers), BorgBackup, Burp, Duplicati, Duplicity, Restic, and more. Sometimes taking snapshots of VMware environments is part of the solution, and I have experience with Veeam and SolarWinds Virtualization Manager as well as using the Vcenter API to trigger machine images stored on different parts of the architecture. I’ve also rolled out custom but reliable shell scripts to backup systems using rclone or rdiff-backup.
End-User Client Management and Deployment
Keeping user machines, and servers, up to date using or not using management agents. I’m experienced using Chocolatey to update Windows software and topgrade to update Mac and Linux software, as well as MDM solutions. Experience using Clonezilla and Fog for opensource solutions, a custom solution using a PXE server that pushed out ISO imaged, as well as Windows Deployment Services and Microsoft Deployment Toolkit.
Managed Cloud Providers
AWS, Google Cloud, Linode, Digital Ocean, Scaleway, OVH/OVH Cloud, Oracle Cloud, Azure.
OpenStack, Openshift, Eucalyptus (AWS Compatible), and AppScale (Google App Engine Compatible), and multiple VMWare provided products.
Kubernetes, Salt, Terraform, RunDeck, Nomad, and more.
Elasticsearch and the AWS Elasticsearch flavor, Algolia, Azure Search, Yacy, and more.
OS and Process virtualization
Extensive experience with Docker, and using Docker Compose, Docker Swarm, and Portainer to provide a friendly UI. I’ve also worked with LXC directly to keep software on a linux system contained on a kernel level. It’s been a while, but I’ve also worked with OpenVZ before to provide more traditional container-based virtual machines without as much overhead as a traditional VM system. For traditional Virtual Machines, I’ve been an administrator with VCenter and managed everything from machine replication, Vmotion, managing a VMWare vSAN, and VMware Site Recovery Manager.
Ansible, Chef, Puppet, Salt
CI/CD & Application Development
Buildbot, Drone, GitLab CI, GoCD, Kenkins. Capistrano, Fabric, Rocketeer
Network-based distributed Filesystems
Ceph, GlusterFS, Minio, TahoeLAFS, Amazon S3, Cloudflare R2, and Backblaze B2.
Bind, dnsmasq, PowerDNS, extensive knowledge with obscure but important DNS records
FreeRADIUS, OpenLDAP, Active Directory, CardDAV, etc
Elasticsearch, Fluentd, Graylog, Kibana, Logstash, Splunk.
Nagios (and various alternative interfaces and forks like Icinga), Alerta, Cacti, Cabot, Healthchecks, Observium for SNMP monitoriung, Scrutiny for SMART monitoring, Sensu Zabbix, and a few others.
Sometimes there is some overlap, but sometimes you need some more specfics. I’ve previously used Collectd and Statsd, usually writing to InfluxDB with Graphana or Graphite to provide a nice dashboard. I’ve used more integrated solutions like Prometheus
Lots of experience installing, managing, optimizing, monitoring, and securing MySQL/MariaDB, PostgreSQL, Microsoft SQL, and the underrated workhorse SQLite. Traditional experience with InfluxDB, MongoDB, and Key-Value DB’s like Redis and Riak and LevelDB. I have limited installation and operational experience with Cassandra and CouchDB.
Visibility into network device operations, configuration, and ensuring backups is often neglected. I’ve previously rolled a custom Cisco backup bash script, but moved onto using rConfig and RANCID, but now use Oxidized. Big proponent of properly sequestering different categories of devices on different vlans to reduce “leap frog” intrusions.
Several years working with Cisco equipment including core routers, managed switches, and edge routers and VPN ASA’s. Also have experience with Juniper core and edge routers, as well as some low cost small business “barely managed” switches from the like of Dell and HP. I was proficient with Cisco IOS and Junos, but a bit rusty as it’s been a few years. Currently my focus is on software-defined datacenter technologies from providers like VMWare, openStack, and AWS.
Classic experience of managing Windows Remote Desktop servers and Citrix desktop servers. More recently, experience with some basic administration of cloud-based Amazon Workspaces & Azure Virtual Desktops, as well as locally hosted VMWare Horizon desktops.
Email receiving is relatively simple, and I have plenty of experience running Postfix, Sendmail, Exim, and a few others. It’s been several years, but I was formerly an Exchange administrator and migrated all the accounts to Microsoft 365 hosted Exchange and was that administrator. Sending mail is more difficult, but I am familiar with all of the pitfalls like monitoring realtime blackhole lists, setting up a proper DNSSEC DNS record to request reports of emails that are categorized as spam, setting up proper DKIM and SPF records to make sure spammers don’t misappropriate the domain name and that emails cannot be modified in transit because they are digitally signed. I am also experienced managing Google Workspace, and their email offerings.
Office and Work Collaboration Tools:
In addition to the email infrastructure work, I’ve previously installed, maintained, and used various CalDAV and CardDAV implementations to allow for managed and centralized calendars and contact information repositories.
I have also set up various Slack alternatives like Mattermost and Rocket.Chat, and for SaaS solutions like Slack and Discord, I’ve set up custom bots that monitor various rooms and perform custom tasks based on commands issued to the bots. Have configured and administered Matrix, IRC, and XMPP servers and various gateways and bridges.
I was an early adopter of Asterisk, the open source advanced IP-based telephony PBX and VoIP gateway. Since then, I’ve also used openSIPS and FreeSWITCH.
Comfortable building and programming IoT devices using ESP32 or ESP8266 microcontrollers and custom sensors. Routinely build various drones, robots, and other devices.
Built several systems that utilized high-gain WiFi systems, LoRaWAN, Z-Wave, Zigbee, and BT Mesh Networks. Built a small home-rolled OpenRAN 4G/5G mobile system over open airwaves and have experimented with wireless networking using ultrasonics or various light waves.
I have extensive experience in installing, managing, and upgrading Discourse, mastadon, MyBB, phpBB, and a few others that I installed just for the experience.
Content Management Systems
I am well acquainted with enterprise-scale installation and management of Wordpress and Drupal systems, taking installs from single LAMP configurations to multi-server configurations with load balancing, geographic distribution, and server redundancy. I have also set up multiple other systems such as Ghost, Joomla, Magento, and concrete CMS.
Experienced using lots of tools to diagnose system like sysdig as well as deep understanding of what log entries mean and how serious they really are. Adept on analyzing network problems with tools like wireshark and several NetFlow analysis products.
Firm believer that an edge firewall isn’t a replacement for server-level firewalls, keeping services up to date, and ensuring services are configured to use modern security best-practices. Have set up honeypots on networks with critical data to trigger emergency operations when they are compromised. Experience setting up linux servers with full-disk encryption and a micro bootloader with SSH that asks for the decryption key. Also a strong believer in keeping information in databases encrypted whenever possible. Have also set up monitoring of hardware so physical intrusion causes alert broadcast and system shutdown.
Have administered Github Enterprise for several thousand users, as well as a cloud-based Git providers like BitBucket and Gitlab. Also have run smaller internal Git systems such as Gogs and Gitea.
Secure Remote Access
Experience running Cisco ASA’s, as well as custom OpenVPN and WireGuard servers. Also have experience with Cloudflare’s newer offerings like Magic WAN, Magic Firewall, and setting up Network Interconnects and Argo Smart Routing.
Developed and managed small snippets of code that run on SaaS provider edge routers, like Cloudflare Workers, AWS Lambda, Netlift, and Azure Functions. Have also replicated their functions for development and testing purposes using software like blueboat.
Expertise with Debian-based systems, but also adept with RedHat based distributions. Historical experience with OpenBSD and FreeBSD.
Able to configure and manage multi-tenancy environments and isolating users into Linux Jails or individual OpenVZ or Docker containers. Comfortable with the constraints of utilizing the SELinux kernel module, as well as utilizing more esoteric security requirements if necessary, such as port-knocking, moving services to non-standard ports, monitoring user actions to detect potential intrusions, using Web Application Firewalls like ModSecurity or hosted solutions from vendors like Cloudflare or Amazon.
Also experienced using log and IDS monitoring systems to trigger responses to malicious actors or bots from redirecting their traffic to read-only static web servers, serving them false generated data, throttling their speed, or just blocking their IP at the edge.
Experience with IIS, Apache, Nginx, Highttepd, Caddy, and a few esoteric micro web servers for small docker images.
To enhance reliability, I have experience putting multiple servers running the same service behind HAProxy, OpenResty, Traefik, or AWS ELB to allow for dynamically spinning up or down additional back-end servers based on demand. I have also configured load balancers with AnyCast IP’s so traffic is routed to the closest one (from a network perspective). A cheaper solution, but just as effective, is using an AnyCast DNS provider that can be configured to give a different IP address to the load balancer based on the location of the DNS request.
Platform Development and Integration
I have previously done custom coding and integration to tie together systems from SaaS providers to self-hosted systems by getting to know and integrate with their API’s. For instance, Home-Assistant being configured to make alterations to Cloudflare DNS configurations when a person leaves the house, Node-RED being configured to perform local actions when a combination of external factors happen, and writing custom Amazon Alexa routines.
Scripting and Programming
Comfortable writing scripts in various shell formats such as Bash, Fish, sh, csh, etc. Also often write more complicates scripts in PHP (for access to useful composer packages), Perl, or Python. Comfortable integrating those creations into a server properly using SystemD rather than a standard Cron job for better visibility, control, and error handling.
Early remote learning university. Ended a few credits short.