Blocks on the bitcoin blockchain have a maximum size limit. The current single block size limit is 1,000,000 bytes (1 MegaByte), although with the introduction of SegWit some of the data of which transactions are composed will not fit within this limit.
Summary
Virtually unlimited number of Bitcoin transactions
SegWit separates the so-called “witnesses” from the transaction inputs, and only the latter count towards the total block weight in bytes.
This means that it is effectively not possible to include in a block a transaction whose inputs weigh more than 1MB in total, but it is possible to include in a block a transaction whose total weight, including witnesses, is greater, as long as the inputs weigh less than 1MB.
Transactions that are not included in a block are not valid. However, the issue is actually much more complex.
It is worth mentioning that nowadays, BTC transactions may not even be recorded on the blockchain.
Thanks to the second layer solution “Lightning Network“, it is possible to register on the blockchain only the creation and closing transaction of a channel, and then to carry out a virtually unlimited number of transactions with that channel that do not have to be registered on the blockchain.
Since these off-chain transactions do not have to be included in any blockchain, they could even be larger than 1 MB, if sender and receiver accept them. In this case, the software used by the two parties involved in the transaction decides the maximum byte limits of the transaction.
LN is mainly used for small payments, so the issue of transaction size does not arise.
When a BTC transaction exceeds 1MB of input
The weight of a single transaction in bytes is very small, so that in 1 MB there can be more than 3,000, counting only the input. It is not easy to imagine the need to create a transaction that weighs 3,000 times more than the norm, but in theory it would be possible.
Moreover, there is another huge problem.
Transactions actually compete with each other to be included in a block and then confirmed. To do this, their senders raise fees to try to persuade miners to include them in a block.
A transaction with an input weight of 1 MB would have to convince the miner not to include any other transactions in the block, so its fees would have to be higher than what the miner could collect by choosing to include other pending transactions instead.
When it is the transaction sender who mines the block
Given a reward of 6.25 BTC per mined block, the miner rarely collects more than a 0.5 BTC fee. So a 1 MB input transaction would have to offer the miner around $23,000 in bitcoin fees, which is very significant for a single transaction.
However, if the miner is the same sender of the transaction, it may decide to include it in a block even without collecting any fees, as it does not make sense to self-pay.
Large transactions usually require too many fees to be included in a block, unless the sender of the transaction mines the block that includes them. However, as no miner knows exactly when they will be able to mine a block, and as only individual miners with immense amounts of computing power can actually validate a block on their own, this seems particularly remote.
In the future, the 1 MB limit may be extended, but for now, this possibility does not seem to be on the horizon.