Mining ore can be done with hands, but is much more efficiently done with a pickaxe. Mining speed is increased with each successive metal pickaxe upgrade (with the exception of the gold pickaxe which is slow, but has other benefits).
Using gem pickaxes increases your yield. Like metal, these benefits increase with progressively more expensive gem pickaxes. The life of gem pickaxes can be extended by only using them for the final strike; use a metal pickaxe until the block is nearly mined then immediately switch to a gem pickaxe.
Gems are found in caverns. They are picked up by hand, not mined using a pickaxe with the exception of the gold pickaxe which can produce gems, ores, and time crystals with each strike. Since the gold pickaxe can produce bonus resources for each strike, using it only for the final strike (like with gem pickaxes) is only beneficial in prolonging its life. It will not increase yield.
The yield multiplier for gem pickaxes applies to time crystals, so use the best gem pickaxe you have to maximize yield.